[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:54:00 dist-test-slave-k6lz krb5kdc[2784](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:54:00 dist-test-slave-k6lz krb5kdc[2784](info): set up 2 sockets
Sep 28 21:54:00 dist-test-slave-k6lz krb5kdc[2784](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260928 21:54:01.040102 2777 mini_kdc.cc:121] Time spent starting KDC: real 0.965s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Sep 28 21:54:01 dist-test-slave-k6lz krb5kdc[2784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:54:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:54:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab.
Entry for principal kudu/127.2.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab.
I20260928 21:54:01.167994 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:34927
--webserver_interface=127.2.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab
--principal=kudu/127.2.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:35039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.126:34927
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:54:01.254938 2797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:54:01.255175 2797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:54:01.255208 2797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:54:01.256484 2797 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:54:01.256534 2797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:54:01.256554 2797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:54:01.256570 2797 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:54:01.256584 2797 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:54:01.258193 2797 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:35039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.126:34927
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab
--principal=kudu/127.2.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:34927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:54:01.258458 2797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:54:01.258698 2797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:54:01.264163 2803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:01.264207 2805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:01.264416 2797 server_base.cc:1034] running on GCE node
W20260928 21:54:01.264209 2802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:01.264777 2797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:01.265065 2797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:01.266225 2797 hybrid_clock.cc:648] HybridClock initialized: now 1790632441266210 us; error 34 us; skew 500 ppm
Sep 28 21:54:01 dist-test-slave-k6lz krb5kdc[2784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632441, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:54:01.268954 2797 init.cc:377] Logged in from keytab as kudu/127.2.182.126@KRBTEST.COM (short username kudu)
I20260928 21:54:01.269488 2797 webserver.cc:460] Webserver started at http://127.2.182.126:39157/ using document root <none> and password file <none>
I20260928 21:54:01.269694 2797 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:01.269738 2797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:01.269845 2797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:54:01.270910 2797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/minicluster-data/master-0/data/instance:
uuid: "d0f8af394d724355a4acfef58710742d"
format_stamp: "Formatted at 2026-09-28 21:54:01 on dist-test-slave-k6lz"
server_key: "874f00bca3b0c9f75f917946a8364cd3"
server_key_iv: "e699b72cf74c347eed01f1e43a17fb87"
server_key_version: "encryptionkey@0"
I20260928 21:54:01.271256 2797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/minicluster-data/master-0/wal/instance:
uuid: "d0f8af394d724355a4acfef58710742d"
format_stamp: "Formatted at 2026-09-28 21:54:01 on dist-test-slave-k6lz"
server_key: "874f00bca3b0c9f75f917946a8364cd3"
server_key_iv: "e699b72cf74c347eed01f1e43a17fb87"
server_key_version: "encryptionkey@0"
I20260928 21:54:01.272806 2797 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:54:01.273695 2812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:01.273893 2797 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:54:01.273991 2797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/minicluster-data/master-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/minicluster-data/master-0/wal
uuid: "d0f8af394d724355a4acfef58710742d"
format_stamp: "Formatted at 2026-09-28 21:54:01 on dist-test-slave-k6lz"
server_key: "874f00bca3b0c9f75f917946a8364cd3"
server_key_iv: "e699b72cf74c347eed01f1e43a17fb87"
server_key_version: "encryptionkey@0"
I20260928 21:54:01.274056 2797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:01.293084 2797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:54:01.295990 2797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:54:01.296252 2797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:54:01.301149 2797 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.126:34927
I20260928 21:54:01.301230 2864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.126:34927 every 8 connection(s)
I20260928 21:54:01.301739 2797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
I20260928 21:54:01.302495 2865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:54:01.303730 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 2797
I20260928 21:54:01.303850 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/minicluster-data/master-0/wal/instance
I20260928 21:54:01.304646 2777 external_mini_cluster.cc:1428] Setting key ad652a96899ae3dd75bb536c821c66f9
I20260928 21:54:01.305526 2865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0f8af394d724355a4acfef58710742d: Bootstrap starting.
I20260928 21:54:01.306366 2865 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0f8af394d724355a4acfef58710742d: Neither blocks nor log segments found. Creating new log.
I20260928 21:54:01.306710 2865 log.cc:824] T 00000000000000000000000000000000 P d0f8af394d724355a4acfef58710742d: Log is configured to *not* fsync() on all Append() calls
I20260928 21:54:01.307593 2865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0f8af394d724355a4acfef58710742d: No bootstrap required, opened a new log
I20260928 21:54:01.309677 2865 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0f8af394d724355a4acfef58710742d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0f8af394d724355a4acfef58710742d" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 34927 } }
I20260928 21:54:01.309837 2865 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0f8af394d724355a4acfef58710742d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:54:01.309859 2865 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0f8af394d724355a4acfef58710742d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0f8af394d724355a4acfef58710742d, State: Initialized, Role: FOLLOWER
I20260928 21:54:01.310096 2865 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0f8af394d724355a4acfef58710742d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0f8af394d724355a4acfef58710742d" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 34927 } }
I20260928 21:54:01.310204 2865 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0f8af394d724355a4acfef58710742d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:54:01.310230 2865 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0f8af394d724355a4acfef58710742d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:54:01.310294 2865 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0f8af394d724355a4acfef58710742d [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:54:01.311383 2865 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0f8af394d724355a4acfef58710742d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0f8af394d724355a4acfef58710742d" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 34927 } }
I20260928 21:54:01.311489 2865 leader_election.cc:302] T 00000000000000000000000000000000 P d0f8af394d724355a4acfef58710742d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0f8af394d724355a4acfef58710742d; no voters:
I20260928 21:54:01.311822 2865 leader_election.cc:288] T 00000000000000000000000000000000 P d0f8af394d724355a4acfef58710742d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:54:01.311854 2870 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0f8af394d724355a4acfef58710742d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:54:01.312054 2870 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0f8af394d724355a4acfef58710742d [term 1 LEADER]: Becoming Leader. State: Replica: d0f8af394d724355a4acfef58710742d, State: Running, Role: LEADER
I20260928 21:54:01.312172 2865 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0f8af394d724355a4acfef58710742d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:54:01.312281 2870 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0f8af394d724355a4acfef58710742d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0f8af394d724355a4acfef58710742d" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 34927 } }
I20260928 21:54:01.312644 2870 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0f8af394d724355a4acfef58710742d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0f8af394d724355a4acfef58710742d. Latest consensus state: current_term: 1 leader_uuid: "d0f8af394d724355a4acfef58710742d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0f8af394d724355a4acfef58710742d" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 34927 } } }
I20260928 21:54:01.312732 2871 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0f8af394d724355a4acfef58710742d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0f8af394d724355a4acfef58710742d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0f8af394d724355a4acfef58710742d" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 34927 } } }
I20260928 21:54:01.312811 2871 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0f8af394d724355a4acfef58710742d [sys.catalog]: This master's current role is: LEADER
I20260928 21:54:01.313030 2870 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0f8af394d724355a4acfef58710742d [sys.catalog]: This master's current role is: LEADER
I20260928 21:54:01.313328 2877 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:54:01.313805 2877 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:54:01.315433 2877 catalog_manager.cc:1347] Generated new cluster ID: 629d4cf58b644c0b9c955cd3bfa9b01f
I20260928 21:54:01.315500 2877 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 21:54:01 dist-test-slave-k6lz krb5kdc[2784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:54:01.321683 2877 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:54:01.322221 2877 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:54:01.327220 2877 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0f8af394d724355a4acfef58710742d: Generated new TSK 0
I20260928 21:54:01.327523 2877 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:54:01.328451 2868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:01.305948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58906 (local address 127.2.182.126:34927)
0928 21:54:01.306254 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:01.306262 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:01.306514 (+ 252us) server_negotiation.cc:407] Connection header received
0928 21:54:01.309688 (+ 3174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:01.309697 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:01.309764 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:01.310010 (+ 246us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:01.311898 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:01.313499 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:01.314955 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:01.315561 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:01.315677 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:01.321088 (+ 5411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:01.321094 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:01.321095 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:01.321111 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:01.324535 (+ 3424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:01.325607 (+ 1072us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:01.325614 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:01.325614 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:01.325784 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:01.326550 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:01.326553 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:01.326554 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:01.326979 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:01.327318 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:01.327993 (+ 675us) server_negotiation.cc:299] Negotiation successful
0928 21:54:01.328217 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":95,"threads_started":1}
WARNING: no policy specified for kudu/127.2.182.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab.
Entry for principal kudu/127.2.182.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab.
I20260928 21:54:01.364961 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.65:0
--local_ip_for_outbound_sockets=127.2.182.65
--webserver_interface=127.2.182.65
--webserver_port=0
--tserver_master_addrs=127.2.182.126:34927
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab
--principal=kudu/127.2.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:35039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:54:01.461376 2891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:54:01.461587 2891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:54:01.461613 2891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:54:01.462813 2891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:54:01.462855 2891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:54:01.462882 2891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.65
I20260928 21:54:01.464430 2891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:35039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab
--principal=kudu/127.2.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.182.65
--webserver_port=0
--tserver_master_addrs=127.2.182.126:34927
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2891
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.65
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:54:01.464639 2891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:54:01.464843 2891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:54:01.469998 2896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:01.470002 2897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:01.470033 2891 server_base.cc:1034] running on GCE node
W20260928 21:54:01.470002 2899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:01.470443 2891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:01.470695 2891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:01.471850 2891 hybrid_clock.cc:648] HybridClock initialized: now 1790632441471836 us; error 30 us; skew 500 ppm
Sep 28 21:54:01 dist-test-slave-k6lz krb5kdc[2784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632441, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:54:01.474774 2891 init.cc:377] Logged in from keytab as kudu/127.2.182.65@KRBTEST.COM (short username kudu)
I20260928 21:54:01.475296 2891 webserver.cc:460] Webserver started at http://127.2.182.65:33597/ using document root <none> and password file <none>
I20260928 21:54:01.475550 2891 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:01.475622 2891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:01.475746 2891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:54:01.476852 2891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/minicluster-data/ts-0/data/instance:
uuid: "1e11651a1cce4e6d81f9f2597dab25ee"
format_stamp: "Formatted at 2026-09-28 21:54:01 on dist-test-slave-k6lz"
server_key: "17550bd0931c1f65d332e458b4415c55"
server_key_iv: "ad392f40048626ac6defdc44c08afe0c"
server_key_version: "encryptionkey@0"
I20260928 21:54:01.477388 2891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/minicluster-data/ts-0/wal/instance:
uuid: "1e11651a1cce4e6d81f9f2597dab25ee"
format_stamp: "Formatted at 2026-09-28 21:54:01 on dist-test-slave-k6lz"
server_key: "17550bd0931c1f65d332e458b4415c55"
server_key_iv: "ad392f40048626ac6defdc44c08afe0c"
server_key_version: "encryptionkey@0"
I20260928 21:54:01.478997 2891 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 21:54:01.480015 2906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:01.480284 2891 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:54:01.480391 2891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/minicluster-data/ts-0/wal
uuid: "1e11651a1cce4e6d81f9f2597dab25ee"
format_stamp: "Formatted at 2026-09-28 21:54:01 on dist-test-slave-k6lz"
server_key: "17550bd0931c1f65d332e458b4415c55"
server_key_iv: "ad392f40048626ac6defdc44c08afe0c"
server_key_version: "encryptionkey@0"
I20260928 21:54:01.480464 2891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:01.509342 2891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:54:01.511943 2891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:54:01.512095 2891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:54:01.512351 2891 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:54:01.512753 2891 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:54:01.512856 2891 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:01.512915 2891 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:54:01.512934 2891 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:01.518497 2891 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.65:34909
I20260928 21:54:01.518543 3019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.65:34909 every 8 connection(s)
I20260928 21:54:01.518919 2891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
I20260928 21:54:01.520722 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 2891
I20260928 21:54:01.520812 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/minicluster-data/ts-0/wal/instance
I20260928 21:54:01.520968 2777 external_mini_cluster.cc:1428] Setting key 3d7f21fab936354ff918ce729e6b767f
Sep 28 21:54:01 dist-test-slave-k6lz krb5kdc[2784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632441, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.65@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.182.66@KRBTEST.COM; defaulting to no policy
I20260928 21:54:01.529531 2868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:01.519814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.65:37853 (local address 127.2.182.126:34927)
0928 21:54:01.519894 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:01.519896 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:01.520034 (+ 138us) server_negotiation.cc:407] Connection header received
0928 21:54:01.520557 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:01.520558 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:01.520572 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:01.520631 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:01.521790 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:01.522281 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:01.523546 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:01.523851 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:01.523923 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:01.526584 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:01.526586 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:01.526586 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:01.526593 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:01.527975 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:01.528464 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:01.528466 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:01.528466 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:01.528520 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:01.528788 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:01.528789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:01.528789 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:01.528921 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:01.528987 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:01.529334 (+ 347us) server_negotiation.cc:299] Negotiation successful
0928 21:54:01.529422 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 21:54:01.529700 3022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:01.519758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:34927 (local address 127.2.182.65:37853)
0928 21:54:01.519912 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:01.519950 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:01.520427 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:01.520741 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:01.520746 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:01.521447 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:01.521689 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:01.521696 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:01.522363 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:01.522364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:01.523467 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:01.523471 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:01.523954 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:01.523956 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:01.524171 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:01.524709 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:01.524720 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:01.526460 (+ 1740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:01.528059 (+ 1599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:01.528068 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:01.528071 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:01.528377 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:01.528583 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:01.528584 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:01.528584 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:01.528717 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:01.528987 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:01.528995 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:01.529200 (+ 205us) client_negotiation.cc:769] Sending connection context
0928 21:54:01.529301 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 21:54:01.529472 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":69,"threads_started":1}
I20260928 21:54:01.530362 3020 heartbeater.cc:347] Connected to a master server at 127.2.182.126:34927
I20260928 21:54:01.530460 3020 heartbeater.cc:464] Registering TS with master...
I20260928 21:54:01.530633 3020 heartbeater.cc:511] Master 127.2.182.126:34927 requested a full tablet report, sending...
I20260928 21:54:01.531308 2829 ts_manager.cc:194] Registered new tserver with Master: 1e11651a1cce4e6d81f9f2597dab25ee (127.2.182.65:34909)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.66@KRBTEST.COM" created.
I20260928 21:54:01.532047 2829 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.65@KRBTEST.COM'} at 127.2.182.65:37853
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab.
Entry for principal kudu/127.2.182.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab.
I20260928 21:54:01.547025 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.66:0
--local_ip_for_outbound_sockets=127.2.182.66
--webserver_interface=127.2.182.66
--webserver_port=0
--tserver_master_addrs=127.2.182.126:34927
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab
--principal=kudu/127.2.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:35039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:54:01.627799 3025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:54:01.628017 3025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:54:01.628042 3025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:54:01.629256 3025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:54:01.629302 3025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:54:01.629333 3025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.66
I20260928 21:54:01.630854 3025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:35039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab
--principal=kudu/127.2.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.182.66
--webserver_port=0
--tserver_master_addrs=127.2.182.126:34927
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.66
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:54:01.631098 3025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:54:01.631323 3025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:54:01.636716 3030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:01.636754 3033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:01.636689 3031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:01.636754 3025 server_base.cc:1034] running on GCE node
I20260928 21:54:01.637152 3025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:01.637404 3025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:01.638567 3025 hybrid_clock.cc:648] HybridClock initialized: now 1790632441638553 us; error 29 us; skew 500 ppm
Sep 28 21:54:01 dist-test-slave-k6lz krb5kdc[2784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632441, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:54:01.648581 3025 init.cc:377] Logged in from keytab as kudu/127.2.182.66@KRBTEST.COM (short username kudu)
I20260928 21:54:01.649040 3025 webserver.cc:460] Webserver started at http://127.2.182.66:32801/ using document root <none> and password file <none>
I20260928 21:54:01.649281 3025 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:01.649327 3025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:01.649441 3025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:54:01.650522 3025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/minicluster-data/ts-1/data/instance:
uuid: "3a3f9c79fece41c5a80359ee9316e236"
format_stamp: "Formatted at 2026-09-28 21:54:01 on dist-test-slave-k6lz"
server_key: "4a378cee93cd2d60c007e2d67ca3caff"
server_key_iv: "3eb3e1515359272cafcbf6cbde3ee1a1"
server_key_version: "encryptionkey@0"
I20260928 21:54:01.650848 3025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/minicluster-data/ts-1/wal/instance:
uuid: "3a3f9c79fece41c5a80359ee9316e236"
format_stamp: "Formatted at 2026-09-28 21:54:01 on dist-test-slave-k6lz"
server_key: "4a378cee93cd2d60c007e2d67ca3caff"
server_key_iv: "3eb3e1515359272cafcbf6cbde3ee1a1"
server_key_version: "encryptionkey@0"
I20260928 21:54:01.652303 3025 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 21:54:01.653215 3040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:01.653465 3025 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:54:01.653542 3025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/minicluster-data/ts-1/wal
uuid: "3a3f9c79fece41c5a80359ee9316e236"
format_stamp: "Formatted at 2026-09-28 21:54:01 on dist-test-slave-k6lz"
server_key: "4a378cee93cd2d60c007e2d67ca3caff"
server_key_iv: "3eb3e1515359272cafcbf6cbde3ee1a1"
server_key_version: "encryptionkey@0"
I20260928 21:54:01.653627 3025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:01.679060 3025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:54:01.681512 3025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:54:01.681754 3025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:54:01.681993 3025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:54:01.682335 3025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:54:01.682372 3025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:01.682394 3025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:54:01.682404 3025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:01.689421 3025 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.66:45513
I20260928 21:54:01.689461 3153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.66:45513 every 8 connection(s)
I20260928 21:54:01.689813 3025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
I20260928 21:54:01.692330 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 3025
I20260928 21:54:01.692413 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/minicluster-data/ts-1/wal/instance
I20260928 21:54:01.692548 2777 external_mini_cluster.cc:1428] Setting key 601da6c4b9e7074aea2dc8fc5689e0d5
Sep 28 21:54:01 dist-test-slave-k6lz krb5kdc[2784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632441, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.66@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.182.67@KRBTEST.COM; defaulting to no policy
I20260928 21:54:01.700834 2868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:01.690627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.66:57555 (local address 127.2.182.126:34927)
0928 21:54:01.690825 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:01.690827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:01.690997 (+ 170us) server_negotiation.cc:407] Connection header received
0928 21:54:01.691584 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:01.691585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:01.691597 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:01.691653 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:01.692685 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:01.693367 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:01.694416 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:01.694611 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:01.694672 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:01.697792 (+ 3120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:01.697793 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:01.697794 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:01.697801 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:01.699209 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:01.699737 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:01.699739 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:01.699739 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:01.699798 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:01.700084 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:01.700085 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:01.700085 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:01.700258 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:01.700312 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:01.700684 (+ 372us) server_negotiation.cc:299] Negotiation successful
0928 21:54:01.700761 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164}
I20260928 21:54:01.701058 3156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:01.690546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:34927 (local address 127.2.182.66:57555)
0928 21:54:01.690875 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:01.690915 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:01.691489 (+ 574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:01.691717 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:01.691721 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:01.692344 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:01.692580 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:01.692587 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:01.693445 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:01.693446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:01.694304 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:01.694306 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:01.694689 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:01.694690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:01.694828 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:01.695917 (+ 1089us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:01.695925 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:01.697656 (+ 1731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:01.699299 (+ 1643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:01.699309 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:01.699313 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:01.699645 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:01.699866 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:01.699867 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:01.699867 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:01.700015 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:01.700327 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:01.700334 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:01.700525 (+ 191us) client_negotiation.cc:769] Sending connection context
0928 21:54:01.700635 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 21:54:01.700812 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":101,"threads_started":1}
I20260928 21:54:01.701761 3154 heartbeater.cc:347] Connected to a master server at 127.2.182.126:34927
I20260928 21:54:01.701870 3154 heartbeater.cc:464] Registering TS with master...
I20260928 21:54:01.702047 3154 heartbeater.cc:511] Master 127.2.182.126:34927 requested a full tablet report, sending...
I20260928 21:54:01.702483 2829 ts_manager.cc:194] Registered new tserver with Master: 3a3f9c79fece41c5a80359ee9316e236 (127.2.182.66:45513)
I20260928 21:54:01.702900 2829 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.66@KRBTEST.COM'} at 127.2.182.66:57555
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab.
Entry for principal kudu/127.2.182.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab.
I20260928 21:54:01.719090 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.67:0
--local_ip_for_outbound_sockets=127.2.182.67
--webserver_interface=127.2.182.67
--webserver_port=0
--tserver_master_addrs=127.2.182.126:34927
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab
--principal=kudu/127.2.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:35039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:54:01.801636 3159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:54:01.801862 3159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:54:01.801887 3159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:54:01.803076 3159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:54:01.803112 3159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:54:01.803138 3159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.67
I20260928 21:54:01.804685 3159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:35039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab
--principal=kudu/127.2.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.182.67
--webserver_port=0
--tserver_master_addrs=127.2.182.126:34927
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.67
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:54:01.804908 3159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:54:01.805099 3159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:54:01.810207 3164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:01.810217 3165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:01.810228 3167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:01.810564 3159 server_base.cc:1034] running on GCE node
I20260928 21:54:01.810793 3159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:01.811012 3159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:01.812156 3159 hybrid_clock.cc:648] HybridClock initialized: now 1790632441812143 us; error 30 us; skew 500 ppm
Sep 28 21:54:01 dist-test-slave-k6lz krb5kdc[2784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632441, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:54:01.814916 3159 init.cc:377] Logged in from keytab as kudu/127.2.182.67@KRBTEST.COM (short username kudu)
I20260928 21:54:01.815418 3159 webserver.cc:460] Webserver started at http://127.2.182.67:35325/ using document root <none> and password file <none>
I20260928 21:54:01.815630 3159 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:01.815673 3159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:01.815800 3159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:54:01.816864 3159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/minicluster-data/ts-2/data/instance:
uuid: "14d5fa0a870b4254b86318ccf5ecdaf7"
format_stamp: "Formatted at 2026-09-28 21:54:01 on dist-test-slave-k6lz"
server_key: "99f14cc0b01985f260428524b28de285"
server_key_iv: "8a412d73e3bde385dbcf4fd873991009"
server_key_version: "encryptionkey@0"
I20260928 21:54:01.817212 3159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/minicluster-data/ts-2/wal/instance:
uuid: "14d5fa0a870b4254b86318ccf5ecdaf7"
format_stamp: "Formatted at 2026-09-28 21:54:01 on dist-test-slave-k6lz"
server_key: "99f14cc0b01985f260428524b28de285"
server_key_iv: "8a412d73e3bde385dbcf4fd873991009"
server_key_version: "encryptionkey@0"
I20260928 21:54:01.818545 3159 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 21:54:01.819352 3174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:01.819528 3159 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:54:01.819628 3159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/minicluster-data/ts-2/wal
uuid: "14d5fa0a870b4254b86318ccf5ecdaf7"
format_stamp: "Formatted at 2026-09-28 21:54:01 on dist-test-slave-k6lz"
server_key: "99f14cc0b01985f260428524b28de285"
server_key_iv: "8a412d73e3bde385dbcf4fd873991009"
server_key_version: "encryptionkey@0"
I20260928 21:54:01.819777 3159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:01.826157 3159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:54:01.828284 3159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:54:01.828418 3159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:54:01.828622 3159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:54:01.828920 3159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:54:01.828959 3159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:01.828987 3159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:54:01.829005 3159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:01.835391 3159 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.67:36231
I20260928 21:54:01.835402 3287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.67:36231 every 8 connection(s)
I20260928 21:54:01.835753 3159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
Sep 28 21:54:01 dist-test-slave-k6lz krb5kdc[2784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632441, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.67@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:54:01.844099 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 3159
I20260928 21:54:01.844184 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632440055392-2777-0/minicluster-data/ts-2/wal/instance
I20260928 21:54:01.844374 2777 external_mini_cluster.cc:1428] Setting key b3db66ea9a33afd84a68af0e98a7c8af
I20260928 21:54:01.845296 2868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:01.836529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.67:59843 (local address 127.2.182.126:34927)
0928 21:54:01.836582 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:01.836585 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:01.836731 (+ 146us) server_negotiation.cc:407] Connection header received
0928 21:54:01.837251 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:01.837252 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:01.837264 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:01.837327 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:01.838242 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:01.838833 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:01.840059 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:01.840273 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:01.840345 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:01.842482 (+ 2137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:01.842484 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:01.842484 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:01.842492 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:01.843720 (+ 1228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:01.844228 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:01.844229 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:01.844229 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:01.844297 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:01.844557 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:01.844558 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:01.844558 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:01.844676 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:01.844799 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:01.845123 (+ 324us) server_negotiation.cc:299] Negotiation successful
0928 21:54:01.845207 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 21:54:01.845445 3290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:01.836458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:34927 (local address 127.2.182.67:59843)
0928 21:54:01.836607 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:01.836649 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:01.837122 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:01.837380 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:01.837383 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:01.837942 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:01.838134 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:01.838138 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:01.838943 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:01.838944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:01.839953 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:01.839956 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:01.840376 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:01.840378 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:01.840501 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:01.841047 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:01.841055 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:01.842383 (+ 1328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:01.843851 (+ 1468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:01.843857 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:01.843859 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:01.844142 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:01.844361 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:01.844362 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:01.844362 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:01.844495 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:01.844725 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:01.844731 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:01.844966 (+ 235us) client_negotiation.cc:769] Sending connection context
0928 21:54:01.845078 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 21:54:01.845245 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":61,"threads_started":1}
I20260928 21:54:01.845916 3288 heartbeater.cc:347] Connected to a master server at 127.2.182.126:34927
I20260928 21:54:01.846004 3288 heartbeater.cc:464] Registering TS with master...
I20260928 21:54:01.846174 3288 heartbeater.cc:511] Master 127.2.182.126:34927 requested a full tablet report, sending...
I20260928 21:54:01.846534 2829 ts_manager.cc:194] Registered new tserver with Master: 14d5fa0a870b4254b86318ccf5ecdaf7 (127.2.182.67:36231)
I20260928 21:54:01.846994 2829 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.67@KRBTEST.COM'} at 127.2.182.67:59843
I20260928 21:54:01.856936 2777 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 21:54:01 dist-test-slave-k6lz krb5kdc[2784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632441, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 21:54:01 dist-test-slave-k6lz krb5kdc[2784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632441, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.182.65@KRBTEST.COM
I20260928 21:54:01.881736 3295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:01.872588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52020 (local address 127.2.182.65:34909)
0928 21:54:01.872815 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:01.872820 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:01.872832 (+ 12us) server_negotiation.cc:407] Connection header received
0928 21:54:01.872858 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:01.872860 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:01.872874 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:01.872950 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:01.873601 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:01.874359 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:01.875294 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:01.875686 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:01.875760 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:01.878607 (+ 2847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:01.878616 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:01.878617 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:01.878625 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:01.880455 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:01.880879 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:01.880882 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:01.880882 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:01.880942 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:01.881194 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:01.881196 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:01.881196 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:01.881355 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:01.881508 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:01.881551 (+ 43us) server_negotiation.cc:299] Negotiation successful
0928 21:54:01.881622 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":102,"threads_started":1}
W20260928 21:54:01.882148 2934 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52020
Sep 28 21:54:01 dist-test-slave-k6lz krb5kdc[2784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 21:54:01 dist-test-slave-k6lz krb5kdc[2784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.182.65@KRBTEST.COM
I20260928 21:54:01.905133 3295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:01.897610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52032 (local address 127.2.182.65:34909)
0928 21:54:01.897730 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:01.897732 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:01.897743 (+ 11us) server_negotiation.cc:407] Connection header received
0928 21:54:01.897787 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:01.897788 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:01.897797 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:01.897837 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:01.898436 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:01.899016 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:01.899861 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:01.900065 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:01.900132 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:01.902449 (+ 2317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:01.902451 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:01.902451 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:01.902459 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:01.903955 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:01.904293 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:01.904294 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:01.904295 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:01.904347 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:01.904578 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:01.904579 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:01.904579 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:01.904776 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:01.904881 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:01.904988 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 21:54:01.905058 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260928 21:54:01.905742 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 2891
I20260928 21:54:01.909986 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 3025
I20260928 21:54:01.915388 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 3159
I20260928 21:54:01.919260 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 2797
2026-09-28T21:54:01Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1869 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:54:01 dist-test-slave-k6lz krb5kdc[3302](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:54:01 dist-test-slave-k6lz krb5kdc[3302](info): set up 2 sockets
Sep 28 21:54:01 dist-test-slave-k6lz krb5kdc[3302](info): commencing operation
krb5kdc: starting...
W20260928 21:54:02.894853 2777 mini_kdc.cc:121] Time spent starting KDC: real 0.956s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Sep 28 21:54:02 dist-test-slave-k6lz krb5kdc[3302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:54:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:54:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab.
Entry for principal kudu/127.2.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab.
I20260928 21:54:03.018568 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:46561
--webserver_interface=127.2.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab
--principal=kudu/127.2.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:46429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.126:46561
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:54:03.102350 3315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:54:03.102591 3315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:54:03.102615 3315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:54:03.103986 3315 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:54:03.104038 3315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:54:03.104051 3315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:54:03.104063 3315 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:54:03.104074 3315 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:54:03.105710 3315 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:46429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.126:46561
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab
--principal=kudu/127.2.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:46561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:54:03.105973 3315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:54:03.106199 3315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:54:03.111183 3321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:03.111181 3323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:03.111287 3315 server_base.cc:1034] running on GCE node
W20260928 21:54:03.111199 3320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:03.111599 3315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:03.111873 3315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:03.113009 3315 hybrid_clock.cc:648] HybridClock initialized: now 1790632443112992 us; error 36 us; skew 500 ppm
Sep 28 21:54:03 dist-test-slave-k6lz krb5kdc[3302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632443, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:54:03.115818 3315 init.cc:377] Logged in from keytab as kudu/127.2.182.126@KRBTEST.COM (short username kudu)
I20260928 21:54:03.116305 3315 webserver.cc:460] Webserver started at http://127.2.182.126:46639/ using document root <none> and password file <none>
I20260928 21:54:03.116526 3315 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:03.116577 3315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:03.116703 3315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:54:03.117821 3315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/minicluster-data/master-0/data/instance:
uuid: "2e994df717024fdb9ccc896853fe2c38"
format_stamp: "Formatted at 2026-09-28 21:54:03 on dist-test-slave-k6lz"
server_key: "ca6191d76f5945d70e26dff1d63455ed"
server_key_iv: "41a2c4c946988a678ae26cc35b7b9979"
server_key_version: "encryptionkey@0"
I20260928 21:54:03.118177 3315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/minicluster-data/master-0/wal/instance:
uuid: "2e994df717024fdb9ccc896853fe2c38"
format_stamp: "Formatted at 2026-09-28 21:54:03 on dist-test-slave-k6lz"
server_key: "ca6191d76f5945d70e26dff1d63455ed"
server_key_iv: "41a2c4c946988a678ae26cc35b7b9979"
server_key_version: "encryptionkey@0"
I20260928 21:54:03.119570 3315 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:54:03.120422 3330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:03.120600 3315 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:54:03.120697 3315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/minicluster-data/master-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/minicluster-data/master-0/wal
uuid: "2e994df717024fdb9ccc896853fe2c38"
format_stamp: "Formatted at 2026-09-28 21:54:03 on dist-test-slave-k6lz"
server_key: "ca6191d76f5945d70e26dff1d63455ed"
server_key_iv: "41a2c4c946988a678ae26cc35b7b9979"
server_key_version: "encryptionkey@0"
I20260928 21:54:03.120785 3315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:03.146870 3315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:54:03.149046 3315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:54:03.149160 3315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:54:03.152944 3315 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.126:46561
I20260928 21:54:03.153050 3382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.126:46561 every 8 connection(s)
I20260928 21:54:03.153360 3315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
I20260928 21:54:03.154140 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 3315
I20260928 21:54:03.154260 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/minicluster-data/master-0/wal/instance
I20260928 21:54:03.154325 3383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:54:03.154428 2777 external_mini_cluster.cc:1428] Setting key e04bbbfd45736ffd240cf5dbfc1e7fc7
I20260928 21:54:03.157447 3383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e994df717024fdb9ccc896853fe2c38: Bootstrap starting.
I20260928 21:54:03.158221 3383 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e994df717024fdb9ccc896853fe2c38: Neither blocks nor log segments found. Creating new log.
I20260928 21:54:03.158517 3383 log.cc:824] T 00000000000000000000000000000000 P 2e994df717024fdb9ccc896853fe2c38: Log is configured to *not* fsync() on all Append() calls
I20260928 21:54:03.159207 3383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e994df717024fdb9ccc896853fe2c38: No bootstrap required, opened a new log
Sep 28 21:54:03 dist-test-slave-k6lz krb5kdc[3302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:54:03.161617 3383 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e994df717024fdb9ccc896853fe2c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e994df717024fdb9ccc896853fe2c38" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 46561 } }
I20260928 21:54:03.161814 3383 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e994df717024fdb9ccc896853fe2c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:54:03.161871 3383 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e994df717024fdb9ccc896853fe2c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e994df717024fdb9ccc896853fe2c38, State: Initialized, Role: FOLLOWER
I20260928 21:54:03.162014 3383 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e994df717024fdb9ccc896853fe2c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e994df717024fdb9ccc896853fe2c38" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 46561 } }
I20260928 21:54:03.162091 3383 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e994df717024fdb9ccc896853fe2c38 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:54:03.162122 3383 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e994df717024fdb9ccc896853fe2c38 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:54:03.162178 3383 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e994df717024fdb9ccc896853fe2c38 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:54:03.162739 3383 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e994df717024fdb9ccc896853fe2c38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e994df717024fdb9ccc896853fe2c38" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 46561 } }
I20260928 21:54:03.162837 3383 leader_election.cc:302] T 00000000000000000000000000000000 P 2e994df717024fdb9ccc896853fe2c38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e994df717024fdb9ccc896853fe2c38; no voters:
I20260928 21:54:03.163009 3383 leader_election.cc:288] T 00000000000000000000000000000000 P 2e994df717024fdb9ccc896853fe2c38 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:54:03.163048 3388 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e994df717024fdb9ccc896853fe2c38 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:54:03.163231 3388 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e994df717024fdb9ccc896853fe2c38 [term 1 LEADER]: Becoming Leader. State: Replica: 2e994df717024fdb9ccc896853fe2c38, State: Running, Role: LEADER
I20260928 21:54:03.163283 3383 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e994df717024fdb9ccc896853fe2c38 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:54:03.163355 3388 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e994df717024fdb9ccc896853fe2c38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e994df717024fdb9ccc896853fe2c38" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 46561 } }
I20260928 21:54:03.163705 3388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e994df717024fdb9ccc896853fe2c38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e994df717024fdb9ccc896853fe2c38. Latest consensus state: current_term: 1 leader_uuid: "2e994df717024fdb9ccc896853fe2c38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e994df717024fdb9ccc896853fe2c38" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 46561 } } }
I20260928 21:54:03.163841 3388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e994df717024fdb9ccc896853fe2c38 [sys.catalog]: This master's current role is: LEADER
I20260928 21:54:03.163884 3389 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e994df717024fdb9ccc896853fe2c38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e994df717024fdb9ccc896853fe2c38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e994df717024fdb9ccc896853fe2c38" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 46561 } } }
I20260928 21:54:03.164005 3389 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e994df717024fdb9ccc896853fe2c38 [sys.catalog]: This master's current role is: LEADER
I20260928 21:54:03.164095 3394 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:54:03.164624 3394 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:54:03.165503 3386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:03.155014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39646 (local address 127.2.182.126:46561)
0928 21:54:03.155162 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:03.155170 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:03.155197 (+ 27us) server_negotiation.cc:407] Connection header received
0928 21:54:03.155699 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:03.155713 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:03.155765 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:03.155880 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:03.156766 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:03.157679 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:03.158673 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:03.159085 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:03.159172 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:03.161787 (+ 2615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:03.161792 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:03.161792 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:03.161805 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:03.163837 (+ 2032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:03.164298 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:03.164302 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:03.164302 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:03.164409 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:03.164630 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:03.164631 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:03.164631 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:03.164987 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:03.165105 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:03.165256 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 21:54:03.165362 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":57,"threads_started":1}
I20260928 21:54:03.166599 3394 catalog_manager.cc:1347] Generated new cluster ID: 60c551664db84610b512199416d0f710
I20260928 21:54:03.166656 3394 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:54:03.190341 3394 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:54:03.190918 3394 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:54:03.199712 3394 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e994df717024fdb9ccc896853fe2c38: Generated new TSK 0
I20260928 21:54:03.199945 3394 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.182.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab.
Entry for principal kudu/127.2.182.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab.
I20260928 21:54:03.235563 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.65:0
--local_ip_for_outbound_sockets=127.2.182.65
--webserver_interface=127.2.182.65
--webserver_port=0
--tserver_master_addrs=127.2.182.126:46561
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab
--principal=kudu/127.2.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:46429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:54:03.319866 3409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:54:03.320092 3409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:54:03.320127 3409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:54:03.321368 3409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:54:03.321412 3409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:54:03.321451 3409 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.65
I20260928 21:54:03.322995 3409 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:46429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab
--principal=kudu/127.2.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.182.65
--webserver_port=0
--tserver_master_addrs=127.2.182.126:46561
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.65
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:54:03.323244 3409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:54:03.323511 3409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:54:03.329231 3415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:03.329320 3417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:03.329413 3414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:03.329455 3409 server_base.cc:1034] running on GCE node
I20260928 21:54:03.329721 3409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:03.329978 3409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:03.331144 3409 hybrid_clock.cc:648] HybridClock initialized: now 1790632443331126 us; error 36 us; skew 500 ppm
Sep 28 21:54:03 dist-test-slave-k6lz krb5kdc[3302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632443, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:54:03.333825 3409 init.cc:377] Logged in from keytab as kudu/127.2.182.65@KRBTEST.COM (short username kudu)
I20260928 21:54:03.334299 3409 webserver.cc:460] Webserver started at http://127.2.182.65:33565/ using document root <none> and password file <none>
I20260928 21:54:03.334506 3409 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:03.334553 3409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:03.334679 3409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:54:03.335834 3409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/minicluster-data/ts-0/data/instance:
uuid: "87b57a8ae47b4bf180b95361f946a3d2"
format_stamp: "Formatted at 2026-09-28 21:54:03 on dist-test-slave-k6lz"
server_key: "4396aed799c3bf138a5b398f0a4562e4"
server_key_iv: "211d718950fe48debb5e74768a4dd6c1"
server_key_version: "encryptionkey@0"
I20260928 21:54:03.336163 3409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/minicluster-data/ts-0/wal/instance:
uuid: "87b57a8ae47b4bf180b95361f946a3d2"
format_stamp: "Formatted at 2026-09-28 21:54:03 on dist-test-slave-k6lz"
server_key: "4396aed799c3bf138a5b398f0a4562e4"
server_key_iv: "211d718950fe48debb5e74768a4dd6c1"
server_key_version: "encryptionkey@0"
I20260928 21:54:03.337482 3409 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:54:03.338387 3424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:03.338559 3409 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:54:03.338647 3409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/minicluster-data/ts-0/wal
uuid: "87b57a8ae47b4bf180b95361f946a3d2"
format_stamp: "Formatted at 2026-09-28 21:54:03 on dist-test-slave-k6lz"
server_key: "4396aed799c3bf138a5b398f0a4562e4"
server_key_iv: "211d718950fe48debb5e74768a4dd6c1"
server_key_version: "encryptionkey@0"
I20260928 21:54:03.338719 3409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:03.358932 3409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:54:03.361284 3409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:54:03.361431 3409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:54:03.361691 3409 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:54:03.362022 3409 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:54:03.362057 3409 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:03.362087 3409 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:54:03.362105 3409 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:03.368074 3409 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.65:42055
I20260928 21:54:03.368127 3537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.65:42055 every 8 connection(s)
I20260928 21:54:03.368465 3409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
I20260928 21:54:03.371354 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 3409
I20260928 21:54:03.371444 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/minicluster-data/ts-0/wal/instance
I20260928 21:54:03.371598 2777 external_mini_cluster.cc:1428] Setting key 69bc84fdb3e99539a07113a5206f48ce
Sep 28 21:54:03 dist-test-slave-k6lz krb5kdc[3302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632443, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.65@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:54:03.379299 3386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:03.369268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.65:38557 (local address 127.2.182.126:46561)
0928 21:54:03.369326 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:03.369328 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:03.369500 (+ 172us) server_negotiation.cc:407] Connection header received
0928 21:54:03.370456 (+ 956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:03.370457 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:03.370471 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:03.370529 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:03.371172 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:03.371819 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:03.372899 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:03.373278 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:03.373369 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:03.376115 (+ 2746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:03.376117 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:03.376118 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:03.376127 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:03.377699 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:03.378203 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:03.378204 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:03.378205 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:03.378276 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:03.378536 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:03.378537 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:03.378538 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:03.378704 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:03.378794 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:03.379099 (+ 305us) server_negotiation.cc:299] Negotiation successful
0928 21:54:03.379195 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 21:54:03.379438 3540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:03.369232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:46561 (local address 127.2.182.65:38557)
0928 21:54:03.369376 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:03.369418 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:03.370335 (+ 917us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:03.370602 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:03.370607 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:03.370871 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:03.371069 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:03.371074 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:03.371902 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:03.371903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:03.372786 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:03.372788 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:03.373402 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:03.373404 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:03.373619 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:03.374213 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:03.374223 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:03.376001 (+ 1778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:03.377856 (+ 1855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:03.377864 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:03.377866 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:03.378123 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:03.378350 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:03.378351 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:03.378351 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:03.378472 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:03.378793 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:03.378799 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:03.378963 (+ 164us) client_negotiation.cc:769] Sending connection context
0928 21:54:03.379056 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 21:54:03.379193 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":69,"threads_started":1}
I20260928 21:54:03.380015 3538 heartbeater.cc:347] Connected to a master server at 127.2.182.126:46561
I20260928 21:54:03.380116 3538 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.2.182.66@KRBTEST.COM; defaulting to no policy
I20260928 21:54:03.380308 3538 heartbeater.cc:511] Master 127.2.182.126:46561 requested a full tablet report, sending...
I20260928 21:54:03.381001 3347 ts_manager.cc:194] Registered new tserver with Master: 87b57a8ae47b4bf180b95361f946a3d2 (127.2.182.65:42055)
I20260928 21:54:03.381672 3347 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.65@KRBTEST.COM'} at 127.2.182.65:38557
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab.
Entry for principal kudu/127.2.182.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab.
I20260928 21:54:03.403286 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.66:0
--local_ip_for_outbound_sockets=127.2.182.66
--webserver_interface=127.2.182.66
--webserver_port=0
--tserver_master_addrs=127.2.182.126:46561
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab
--principal=kudu/127.2.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:46429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:54:03.484999 3543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:54:03.485229 3543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:54:03.485255 3543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:54:03.486507 3543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:54:03.486562 3543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:54:03.486599 3543 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.66
I20260928 21:54:03.488207 3543 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:46429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab
--principal=kudu/127.2.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.182.66
--webserver_port=0
--tserver_master_addrs=127.2.182.126:46561
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.66
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:54:03.488454 3543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:54:03.488682 3543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:54:03.493722 3548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:03.493736 3551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:03.493722 3549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:03.494029 3543 server_base.cc:1034] running on GCE node
I20260928 21:54:03.494230 3543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:03.494445 3543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:03.495579 3543 hybrid_clock.cc:648] HybridClock initialized: now 1790632443495566 us; error 25 us; skew 500 ppm
Sep 28 21:54:03 dist-test-slave-k6lz krb5kdc[3302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632443, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:54:03.498260 3543 init.cc:377] Logged in from keytab as kudu/127.2.182.66@KRBTEST.COM (short username kudu)
I20260928 21:54:03.498787 3543 webserver.cc:460] Webserver started at http://127.2.182.66:42263/ using document root <none> and password file <none>
I20260928 21:54:03.499001 3543 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:03.499044 3543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:03.499164 3543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:54:03.500241 3543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/minicluster-data/ts-1/data/instance:
uuid: "93a1c25becc640a2b62bb5cc1cedfeaf"
format_stamp: "Formatted at 2026-09-28 21:54:03 on dist-test-slave-k6lz"
server_key: "db7c99e230bee24f18ff1ee61f49c612"
server_key_iv: "a69e441051ffb380f745670443ecbe34"
server_key_version: "encryptionkey@0"
I20260928 21:54:03.500571 3543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/minicluster-data/ts-1/wal/instance:
uuid: "93a1c25becc640a2b62bb5cc1cedfeaf"
format_stamp: "Formatted at 2026-09-28 21:54:03 on dist-test-slave-k6lz"
server_key: "db7c99e230bee24f18ff1ee61f49c612"
server_key_iv: "a69e441051ffb380f745670443ecbe34"
server_key_version: "encryptionkey@0"
I20260928 21:54:03.501916 3543 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:54:03.502959 3558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:03.503156 3543 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:54:03.503249 3543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/minicluster-data/ts-1/wal
uuid: "93a1c25becc640a2b62bb5cc1cedfeaf"
format_stamp: "Formatted at 2026-09-28 21:54:03 on dist-test-slave-k6lz"
server_key: "db7c99e230bee24f18ff1ee61f49c612"
server_key_iv: "a69e441051ffb380f745670443ecbe34"
server_key_version: "encryptionkey@0"
I20260928 21:54:03.503310 3543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:03.518251 3543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:54:03.520527 3543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:54:03.520655 3543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:54:03.520841 3543 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:54:03.521126 3543 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:54:03.521159 3543 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:03.521181 3543 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:54:03.521193 3543 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:03.526712 3543 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.66:35377
I20260928 21:54:03.526779 3671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.66:35377 every 8 connection(s)
I20260928 21:54:03.527066 3543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
I20260928 21:54:03.528771 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 3543
I20260928 21:54:03.528848 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/minicluster-data/ts-1/wal/instance
I20260928 21:54:03.528993 2777 external_mini_cluster.cc:1428] Setting key f156b3c81a94c86532d534cc3563ec38
Sep 28 21:54:03 dist-test-slave-k6lz krb5kdc[3302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632443, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.66@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.182.67@KRBTEST.COM; defaulting to no policy
I20260928 21:54:03.537375 3386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:03.527809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.66:48417 (local address 127.2.182.126:46561)
0928 21:54:03.527866 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:03.527868 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:03.528049 (+ 181us) server_negotiation.cc:407] Connection header received
0928 21:54:03.529049 (+ 1000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:03.529051 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:03.529064 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:03.529112 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:03.529852 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:03.530335 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:03.531795 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:03.531999 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:03.532060 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:03.534497 (+ 2437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:03.534499 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:03.534499 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:03.534507 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:03.535834 (+ 1327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:03.536320 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:03.536327 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:03.536328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:03.536401 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:03.536745 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:03.536746 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:03.536746 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:03.536867 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:03.536922 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:03.537192 (+ 270us) server_negotiation.cc:299] Negotiation successful
0928 21:54:03.537281 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 21:54:03.537573 3674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:03.527790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:46561 (local address 127.2.182.66:48417)
0928 21:54:03.527932 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:03.527967 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:03.528949 (+ 982us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:03.529245 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:03.529249 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:03.529571 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:03.529774 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:03.529778 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:03.530416 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:03.530417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:03.531705 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:03.531708 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:03.532076 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:03.532077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:03.532230 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:03.532874 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:03.532882 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:03.534385 (+ 1503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:03.535915 (+ 1530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:03.535923 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:03.535926 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:03.536229 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:03.536474 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:03.536476 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:03.536476 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:03.536675 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:03.536924 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:03.536931 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:03.537068 (+ 137us) client_negotiation.cc:769] Sending connection context
0928 21:54:03.537164 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 21:54:03.537376 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":64,"threads_started":1}
I20260928 21:54:03.538125 3672 heartbeater.cc:347] Connected to a master server at 127.2.182.126:46561
I20260928 21:54:03.538239 3672 heartbeater.cc:464] Registering TS with master...
I20260928 21:54:03.538422 3672 heartbeater.cc:511] Master 127.2.182.126:46561 requested a full tablet report, sending...
I20260928 21:54:03.538868 3347 ts_manager.cc:194] Registered new tserver with Master: 93a1c25becc640a2b62bb5cc1cedfeaf (127.2.182.66:35377)
I20260928 21:54:03.539361 3347 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.66@KRBTEST.COM'} at 127.2.182.66:48417
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab.
Entry for principal kudu/127.2.182.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab.
I20260928 21:54:03.554991 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.67:0
--local_ip_for_outbound_sockets=127.2.182.67
--webserver_interface=127.2.182.67
--webserver_port=0
--tserver_master_addrs=127.2.182.126:46561
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab
--principal=kudu/127.2.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:46429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:54:03.636451 3677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:54:03.636699 3677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:54:03.636736 3677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:54:03.638031 3677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:54:03.638078 3677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:54:03.638116 3677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.67
I20260928 21:54:03.639779 3677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:46429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab
--principal=kudu/127.2.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.182.67
--webserver_port=0
--tserver_master_addrs=127.2.182.126:46561
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3677
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.67
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:54:03.640064 3677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:54:03.640285 3677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:54:03.645433 3685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:03.645464 3683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:03.645417 3682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:03.645613 3677 server_base.cc:1034] running on GCE node
I20260928 21:54:03.645835 3677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:03.646202 3677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:03.647395 3677 hybrid_clock.cc:648] HybridClock initialized: now 1790632443647372 us; error 40 us; skew 500 ppm
Sep 28 21:54:03 dist-test-slave-k6lz krb5kdc[3302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632443, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:54:03.650053 3677 init.cc:377] Logged in from keytab as kudu/127.2.182.67@KRBTEST.COM (short username kudu)
I20260928 21:54:03.650542 3677 webserver.cc:460] Webserver started at http://127.2.182.67:40967/ using document root <none> and password file <none>
I20260928 21:54:03.650722 3677 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:03.650772 3677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:03.650893 3677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:54:03.651981 3677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/minicluster-data/ts-2/data/instance:
uuid: "1819f1696bc14160929d3a11c851272a"
format_stamp: "Formatted at 2026-09-28 21:54:03 on dist-test-slave-k6lz"
server_key: "3a4056382824b1f1598f576370f66217"
server_key_iv: "3e83ef8a0426b23f5fecee284cd844c7"
server_key_version: "encryptionkey@0"
I20260928 21:54:03.652272 3677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/minicluster-data/ts-2/wal/instance:
uuid: "1819f1696bc14160929d3a11c851272a"
format_stamp: "Formatted at 2026-09-28 21:54:03 on dist-test-slave-k6lz"
server_key: "3a4056382824b1f1598f576370f66217"
server_key_iv: "3e83ef8a0426b23f5fecee284cd844c7"
server_key_version: "encryptionkey@0"
I20260928 21:54:03.653595 3677 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:54:03.654399 3692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:03.654613 3677 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:54:03.654708 3677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/minicluster-data/ts-2/wal
uuid: "1819f1696bc14160929d3a11c851272a"
format_stamp: "Formatted at 2026-09-28 21:54:03 on dist-test-slave-k6lz"
server_key: "3a4056382824b1f1598f576370f66217"
server_key_iv: "3e83ef8a0426b23f5fecee284cd844c7"
server_key_version: "encryptionkey@0"
I20260928 21:54:03.654778 3677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:03.674238 3677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:54:03.676754 3677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:54:03.676918 3677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:54:03.677278 3677 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:54:03.677639 3677 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:54:03.677677 3677 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:03.677707 3677 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:54:03.677724 3677 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:03.683776 3677 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.67:39045
I20260928 21:54:03.683924 3805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.67:39045 every 8 connection(s)
I20260928 21:54:03.684414 3677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
I20260928 21:54:03.690184 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 3677
I20260928 21:54:03.690286 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632440055392-2777-0/minicluster-data/ts-2/wal/instance
I20260928 21:54:03.690441 2777 external_mini_cluster.cc:1428] Setting key 106a7c12020e9bdb73a57d495adc483d
Sep 28 21:54:03 dist-test-slave-k6lz krb5kdc[3302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632443, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.67@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:54:03.695199 3386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:03.685288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.67:41935 (local address 127.2.182.126:46561)
0928 21:54:03.685428 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:03.685430 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:03.685485 (+ 55us) server_negotiation.cc:407] Connection header received
0928 21:54:03.686491 (+ 1006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:03.686494 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:03.686508 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:03.686555 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:03.687202 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:03.687746 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:03.688729 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:03.688933 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:03.688989 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:03.691593 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:03.691595 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:03.691595 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:03.691604 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:03.693387 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:03.694018 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:03.694019 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:03.694020 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:03.694097 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:03.694432 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:03.694433 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:03.694433 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:03.694550 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:03.694606 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:03.694993 (+ 387us) server_negotiation.cc:299] Negotiation successful
0928 21:54:03.695101 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260928 21:54:03.695420 3808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:03.685201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:46561 (local address 127.2.182.67:41935)
0928 21:54:03.685365 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:03.685410 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:03.686379 (+ 969us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:03.686642 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:03.686646 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:03.686910 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:03.687113 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:03.687118 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:03.687828 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:03.687829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:03.688644 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:03.688646 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:03.689005 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:03.689006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:03.689134 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:03.689683 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:03.689692 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:03.691461 (+ 1769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:03.693470 (+ 2009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:03.693480 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:03.693482 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:03.693937 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:03.694195 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:03.694196 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:03.694196 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:03.694350 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:03.694602 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:03.694624 (+ 22us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:03.694809 (+ 185us) client_negotiation.cc:769] Sending connection context
0928 21:54:03.694962 (+ 153us) client_negotiation.cc:240] Negotiation successful
0928 21:54:03.695145 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":75,"threads_started":1}
I20260928 21:54:03.695858 3806 heartbeater.cc:347] Connected to a master server at 127.2.182.126:46561
I20260928 21:54:03.695952 3806 heartbeater.cc:464] Registering TS with master...
I20260928 21:54:03.696192 3806 heartbeater.cc:511] Master 127.2.182.126:46561 requested a full tablet report, sending...
I20260928 21:54:03.696650 3347 ts_manager.cc:194] Registered new tserver with Master: 1819f1696bc14160929d3a11c851272a (127.2.182.67:39045)
I20260928 21:54:03.697005 3347 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.67@KRBTEST.COM'} at 127.2.182.67:41935
I20260928 21:54:03.702474 2777 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:54:03.709995 3386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:03.704150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39650 (local address 127.2.182.126:46561)
0928 21:54:03.704209 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:03.704211 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:03.704220 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:54:03.704242 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:03.704243 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:03.704253 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:03.704298 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:03.704807 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:03.705300 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:03.705984 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:03.706269 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:03.706324 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:03.707721 (+ 1397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:03.707723 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:03.707723 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:03.707731 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:03.708937 (+ 1206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:03.709261 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:03.709262 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:03.709262 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:03.709319 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:03.709473 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:03.709474 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:03.709475 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:03.709629 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:03.709715 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:03.709827 (+ 112us) server_negotiation.cc:299] Negotiation successful
0928 21:54:03.709903 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 21:54:03.712318 3347 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39650:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:54:03.713905 3347 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:54:03.720556 3819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:03.716591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59438 (local address 127.2.182.65:42055)
0928 21:54:03.716772 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:03.716776 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:03.716877 (+ 101us) server_negotiation.cc:407] Connection header received
0928 21:54:03.716991 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:03.716993 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:03.717008 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:03.717053 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:54:03.717284 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:03.718160 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:03.719390 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:03.720124 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:03.720199 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:03.720263 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:03.720414 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 21:54:03.720463 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":84,"threads_started":1}
I20260928 21:54:03.720577 3822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:03.716869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47916 (local address 127.2.182.66:35377)
0928 21:54:03.717028 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:03.717032 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:03.717040 (+ 8us) server_negotiation.cc:407] Connection header received
0928 21:54:03.717073 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:03.717074 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:03.717086 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:03.717148 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:54:03.717339 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:03.718162 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:03.719636 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:03.720280 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:03.720344 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:03.720399 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:03.720479 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 21:54:03.720513 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":83,"threads_started":1}
I20260928 21:54:03.720598 3821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:03.716689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.66:35377 (local address 127.0.0.1:47916)
0928 21:54:03.716874 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:03.716882 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:03.716924 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:03.717211 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:03.717212 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:03.717214 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:54:03.717270 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:03.717272 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:03.718258 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:03.718259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:03.719560 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:03.719562 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:03.720355 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:03.720356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:03.720417 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:03.720421 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:54:03.720459 (+ 38us) client_negotiation.cc:240] Negotiation successful
0928 21:54:03.720519 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":33,"thread_start_us":58,"threads_started":1}
I20260928 21:54:03.720824 3817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:03.716485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.65:42055 (local address 127.0.0.1:59438)
0928 21:54:03.716809 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:03.716819 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:03.716904 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:03.717150 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:03.717150 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:03.717152 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:54:03.717217 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:03.717220 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:03.718249 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:03.718250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:03.719275 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:03.719278 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:03.720210 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:03.720211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:03.720288 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:03.720304 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 21:54:03.720678 (+ 374us) client_negotiation.cc:240] Negotiation successful
0928 21:54:03.720748 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":109,"thread_start_us":81,"threads_started":1}
I20260928 21:54:03.721604 3818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:03.716477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.67:39045 (local address 127.0.0.1:49198)
0928 21:54:03.716617 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:03.716657 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:03.716717 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:03.717085 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:03.717089 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:03.717091 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:54:03.717217 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:03.717220 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:03.719482 (+ 2262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:03.719483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:03.720558 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:03.720560 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:03.721361 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:03.721362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:03.721429 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:03.721433 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:54:03.721489 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 21:54:03.721526 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":23,"thread_start_us":54,"threads_started":1}
I20260928 21:54:03.721701 3472 tablet_service.cc:1467] Processing CreateTablet for tablet de68ddff5a9a41d0bd35a8c818ff7ba3 (DEFAULT_TABLE table=test-table [id=b61f34fe8c0649289c911843513d34c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:54:03.721850 3606 tablet_service.cc:1467] Processing CreateTablet for tablet de68ddff5a9a41d0bd35a8c818ff7ba3 (DEFAULT_TABLE table=test-table [id=b61f34fe8c0649289c911843513d34c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:54:03.722102 3472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de68ddff5a9a41d0bd35a8c818ff7ba3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:54:03.722168 3606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de68ddff5a9a41d0bd35a8c818ff7ba3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:54:03.724820 3823 tablet_bootstrap.cc:492] T de68ddff5a9a41d0bd35a8c818ff7ba3 P 93a1c25becc640a2b62bb5cc1cedfeaf: Bootstrap starting.
I20260928 21:54:03.725332 3820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:03.716650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49198 (local address 127.2.182.67:39045)
0928 21:54:03.716843 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:03.716846 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:03.716855 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:54:03.716894 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:03.716896 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:03.716911 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:03.716984 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:54:03.718266 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:03.719384 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:03.720641 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:03.721286 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:03.721359 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:03.721415 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:03.725194 (+ 3779us) server_negotiation.cc:299] Negotiation successful
0928 21:54:03.725254 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":73,"threads_started":1}
I20260928 21:54:03.725746 3824 tablet_bootstrap.cc:492] T de68ddff5a9a41d0bd35a8c818ff7ba3 P 87b57a8ae47b4bf180b95361f946a3d2: Bootstrap starting.
I20260928 21:54:03.725818 3823 tablet_bootstrap.cc:654] T de68ddff5a9a41d0bd35a8c818ff7ba3 P 93a1c25becc640a2b62bb5cc1cedfeaf: Neither blocks nor log segments found. Creating new log.
I20260928 21:54:03.726289 3740 tablet_service.cc:1467] Processing CreateTablet for tablet de68ddff5a9a41d0bd35a8c818ff7ba3 (DEFAULT_TABLE table=test-table [id=b61f34fe8c0649289c911843513d34c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:54:03.726491 3823 log.cc:824] T de68ddff5a9a41d0bd35a8c818ff7ba3 P 93a1c25becc640a2b62bb5cc1cedfeaf: Log is configured to *not* fsync() on all Append() calls
I20260928 21:54:03.726596 3824 tablet_bootstrap.cc:654] T de68ddff5a9a41d0bd35a8c818ff7ba3 P 87b57a8ae47b4bf180b95361f946a3d2: Neither blocks nor log segments found. Creating new log.
I20260928 21:54:03.726629 3740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de68ddff5a9a41d0bd35a8c818ff7ba3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:54:03.727078 3824 log.cc:824] T de68ddff5a9a41d0bd35a8c818ff7ba3 P 87b57a8ae47b4bf180b95361f946a3d2: Log is configured to *not* fsync() on all Append() calls
I20260928 21:54:03.727412 3823 tablet_bootstrap.cc:492] T de68ddff5a9a41d0bd35a8c818ff7ba3 P 93a1c25becc640a2b62bb5cc1cedfeaf: No bootstrap required, opened a new log
I20260928 21:54:03.727507 3823 ts_tablet_manager.cc:1397] T de68ddff5a9a41d0bd35a8c818ff7ba3 P 93a1c25becc640a2b62bb5cc1cedfeaf: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 21:54:03.727888 3824 tablet_bootstrap.cc:492] T de68ddff5a9a41d0bd35a8c818ff7ba3 P 87b57a8ae47b4bf180b95361f946a3d2: No bootstrap required, opened a new log
I20260928 21:54:03.728014 3824 ts_tablet_manager.cc:1397] T de68ddff5a9a41d0bd35a8c818ff7ba3 P 87b57a8ae47b4bf180b95361f946a3d2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 21:54:03.729462 3827 tablet_bootstrap.cc:492] T de68ddff5a9a41d0bd35a8c818ff7ba3 P 1819f1696bc14160929d3a11c851272a: Bootstrap starting.
I20260928 21:54:03.730016 3827 tablet_bootstrap.cc:654] T de68ddff5a9a41d0bd35a8c818ff7ba3 P 1819f1696bc14160929d3a11c851272a: Neither blocks nor log segments found. Creating new log.
I20260928 21:54:03.730048 3823 raft_consensus.cc:348] T de68ddff5a9a41d0bd35a8c818ff7ba3 P 93a1c25becc640a2b62bb5cc1cedfeaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87b57a8ae47b4bf180b95361f946a3d2" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 42055 } } peers { permanent_uuid: "1819f1696bc14160929d3a11c851272a" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 39045 } } peers { permanent_uuid: "93a1c25becc640a2b62bb5cc1cedfeaf" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 35377 } }
I20260928 21:54:03.730237 3823 raft_consensus.cc:374] T de68ddff5a9a41d0bd35a8c818ff7ba3 P 93a1c25becc640a2b62bb5cc1cedfeaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:54:03.730212 3824 raft_consensus.cc:348] T de68ddff5a9a41d0bd35a8c818ff7ba3 P 87b57a8ae47b4bf180b95361f946a3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87b57a8ae47b4bf180b95361f946a3d2" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 42055 } } peers { permanent_uuid: "1819f1696bc14160929d3a11c851272a" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 39045 } } peers { permanent_uuid: "93a1c25becc640a2b62bb5cc1cedfeaf" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 35377 } }
I20260928 21:54:03.730319 3823 raft_consensus.cc:729] T de68ddff5a9a41d0bd35a8c818ff7ba3 P 93a1c25becc640a2b62bb5cc1cedfeaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93a1c25becc640a2b62bb5cc1cedfeaf, State: Initialized, Role: FOLLOWER
I20260928 21:54:03.730365 3824 raft_consensus.cc:374] T de68ddff5a9a41d0bd35a8c818ff7ba3 P 87b57a8ae47b4bf180b95361f946a3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:54:03.730432 3824 raft_consensus.cc:729] T de68ddff5a9a41d0bd35a8c818ff7ba3 P 87b57a8ae47b4bf180b95361f946a3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87b57a8ae47b4bf180b95361f946a3d2, State: Initialized, Role: FOLLOWER
I20260928 21:54:03.730468 3823 consensus_queue.cc:260] T de68ddff5a9a41d0bd35a8c818ff7ba3 P 93a1c25becc640a2b62bb5cc1cedfeaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87b57a8ae47b4bf180b95361f946a3d2" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 42055 } } peers { permanent_uuid: "1819f1696bc14160929d3a11c851272a" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 39045 } } peers { permanent_uuid: "93a1c25becc640a2b62bb5cc1cedfeaf" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 35377 } }
I20260928 21:54:03.730538 3824 consensus_queue.cc:260] T de68ddff5a9a41d0bd35a8c818ff7ba3 P 87b57a8ae47b4bf180b95361f946a3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87b57a8ae47b4bf180b95361f946a3d2" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 42055 } } peers { permanent_uuid: "1819f1696bc14160929d3a11c851272a" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 39045 } } peers { permanent_uuid: "93a1c25becc640a2b62bb5cc1cedfeaf" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 35377 } }
I20260928 21:54:03.730558 3827 log.cc:824] T de68ddff5a9a41d0bd35a8c818ff7ba3 P 1819f1696bc14160929d3a11c851272a: Log is configured to *not* fsync() on all Append() calls
I20260928 21:54:03.730791 3823 ts_tablet_manager.cc:1428] T de68ddff5a9a41d0bd35a8c818ff7ba3 P 93a1c25becc640a2b62bb5cc1cedfeaf: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 21:54:03.730811 3672 heartbeater.cc:503] Master 127.2.182.126:46561 was elected leader, sending a full tablet report...
I20260928 21:54:03.730876 3824 ts_tablet_manager.cc:1428] T de68ddff5a9a41d0bd35a8c818ff7ba3 P 87b57a8ae47b4bf180b95361f946a3d2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 21:54:03.730970 3538 heartbeater.cc:503] Master 127.2.182.126:46561 was elected leader, sending a full tablet report...
I20260928 21:54:03.731483 3827 tablet_bootstrap.cc:492] T de68ddff5a9a41d0bd35a8c818ff7ba3 P 1819f1696bc14160929d3a11c851272a: No bootstrap required, opened a new log
I20260928 21:54:03.731552 3827 ts_tablet_manager.cc:1397] T de68ddff5a9a41d0bd35a8c818ff7ba3 P 1819f1696bc14160929d3a11c851272a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 21:54:03.733691 3827 raft_consensus.cc:348] T de68ddff5a9a41d0bd35a8c818ff7ba3 P 1819f1696bc14160929d3a11c851272a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87b57a8ae47b4bf180b95361f946a3d2" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 42055 } } peers { permanent_uuid: "1819f1696bc14160929d3a11c851272a" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 39045 } } peers { permanent_uuid: "93a1c25becc640a2b62bb5cc1cedfeaf" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 35377 } }
I20260928 21:54:03.733880 3827 raft_consensus.cc:374] T de68ddff5a9a41d0bd35a8c818ff7ba3 P 1819f1696bc14160929d3a11c851272a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:54:03.733918 3827 raft_consensus.cc:729] T de68ddff5a9a41d0bd35a8c818ff7ba3 P 1819f1696bc14160929d3a11c851272a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1819f1696bc14160929d3a11c851272a, State: Initialized, Role: FOLLOWER
I20260928 21:54:03.734032 3827 consensus_queue.cc:260] T de68ddff5a9a41d0bd35a8c818ff7ba3 P 1819f1696bc14160929d3a11c851272a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87b57a8ae47b4bf180b95361f946a3d2" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 42055 } } peers { permanent_uuid: "1819f1696bc14160929d3a11c851272a" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 39045 } } peers { permanent_uuid: "93a1c25becc640a2b62bb5cc1cedfeaf" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 35377 } }
I20260928 21:54:03.734279 3827 ts_tablet_manager.cc:1428] T de68ddff5a9a41d0bd35a8c818ff7ba3 P 1819f1696bc14160929d3a11c851272a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 21:54:03.734308 3806 heartbeater.cc:503] Master 127.2.182.126:46561 was elected leader, sending a full tablet report...
W20260928 21:54:03.777613 3673 tablet.cc:2369] T de68ddff5a9a41d0bd35a8c818ff7ba3 P 93a1c25becc640a2b62bb5cc1cedfeaf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:54:03.824699 3831 raft_consensus.cc:482] T de68ddff5a9a41d0bd35a8c818ff7ba3 P 1819f1696bc14160929d3a11c851272a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:54:03.824847 3831 raft_consensus.cc:504] T de68ddff5a9a41d0bd35a8c818ff7ba3 P 1819f1696bc14160929d3a11c851272a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87b57a8ae47b4bf180b95361f946a3d2" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 42055 } } peers { permanent_uuid: "1819f1696bc14160929d3a11c851272a" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 39045 } } peers { permanent_uuid: "93a1c25becc640a2b62bb5cc1cedfeaf" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 35377 } }
I20260928 21:54:03.825155 3831 leader_election.cc:288] T de68ddff5a9a41d0bd35a8c818ff7ba3 P 1819f1696bc14160929d3a11c851272a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87b57a8ae47b4bf180b95361f946a3d2 (127.2.182.65:42055), 93a1c25becc640a2b62bb5cc1cedfeaf (127.2.182.66:35377)
I20260928 21:54:03.828523 3808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:03.825390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.65:42055 (local address 127.2.182.67:60549)
0928 21:54:03.825448 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:03.825461 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:03.825508 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:03.825680 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:03.825681 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:03.825682 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:54:03.825756 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:03.825758 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:03.826564 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:03.826565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:03.827462 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:03.827465 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:03.828189 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:03.828190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:03.828317 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:03.828321 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:54:03.828395 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 21:54:03.828436 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 21:54:03.828523 3819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:03.825466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.67:60549 (local address 127.2.182.65:42055)
0928 21:54:03.825521 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:03.825524 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:03.825535 (+ 11us) server_negotiation.cc:407] Connection header received
0928 21:54:03.825566 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:03.825567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:03.825584 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:03.825658 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:54:03.825910 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:03.826477 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:03.827565 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:03.828084 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:03.828140 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:03.828147 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:03.828422 (+ 275us) server_negotiation.cc:299] Negotiation successful
0928 21:54:03.828462 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 21:54:03.828948 3832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:03.825503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.66:35377 (local address 127.2.182.67:45541)
0928 21:54:03.825697 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:03.825707 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:03.825901 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:03.826021 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:03.826022 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:03.826023 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:54:03.826090 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:03.826091 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:03.826887 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:03.826888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:03.827744 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:03.827746 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:03.828578 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:03.828580 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:03.828751 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:03.828756 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 21:54:03.828847 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 21:54:03.828886 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":124,"threads_started":1}
I20260928 21:54:03.828989 3822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:03.825618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.67:45541 (local address 127.2.182.66:35377)
0928 21:54:03.825663 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:03.825665 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:03.825877 (+ 212us) server_negotiation.cc:407] Connection header received
0928 21:54:03.825969 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:03.825970 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:03.825977 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:03.826012 (+ 35us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:54:03.826150 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:03.826710 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:03.827935 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:03.828437 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:03.828533 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:03.828541 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:03.828880 (+ 339us) server_negotiation.cc:299] Negotiation successful
0928 21:54:03.828926 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 21:54:03.828955 3492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de68ddff5a9a41d0bd35a8c818ff7ba3" candidate_uuid: "1819f1696bc14160929d3a11c851272a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87b57a8ae47b4bf180b95361f946a3d2" is_pre_election: true
I20260928 21:54:03.829123 3492 raft_consensus.cc:2454] T de68ddff5a9a41d0bd35a8c818ff7ba3 P 87b57a8ae47b4bf180b95361f946a3d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1819f1696bc14160929d3a11c851272a in term 0.
I20260928 21:54:03.829311 3626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de68ddff5a9a41d0bd35a8c818ff7ba3" candidate_uuid: "1819f1696bc14160929d3a11c851272a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93a1c25becc640a2b62bb5cc1cedfeaf" is_pre_election: true
I20260928 21:54:03.829341 3695 leader_election.cc:302] T de68ddff5a9a41d0bd35a8c818ff7ba3 P 1819f1696bc14160929d3a11c851272a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1819f1696bc14160929d3a11c851272a, 87b57a8ae47b4bf180b95361f946a3d2; no voters:
I20260928 21:54:03.829438 3626 raft_consensus.cc:2454] T de68ddff5a9a41d0bd35a8c818ff7ba3 P 93a1c25becc640a2b62bb5cc1cedfeaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1819f1696bc14160929d3a11c851272a in term 0.
I20260928 21:54:03.829476 3831 raft_consensus.cc:2781] T de68ddff5a9a41d0bd35a8c818ff7ba3 P 1819f1696bc14160929d3a11c851272a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:54:03.829555 3831 raft_consensus.cc:482] T de68ddff5a9a41d0bd35a8c818ff7ba3 P 1819f1696bc14160929d3a11c851272a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:54:03.829581 3831 raft_consensus.cc:3037] T de68ddff5a9a41d0bd35a8c818ff7ba3 P 1819f1696bc14160929d3a11c851272a [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:54:03.830343 3831 raft_consensus.cc:504] T de68ddff5a9a41d0bd35a8c818ff7ba3 P 1819f1696bc14160929d3a11c851272a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87b57a8ae47b4bf180b95361f946a3d2" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 42055 } } peers { permanent_uuid: "1819f1696bc14160929d3a11c851272a" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 39045 } } peers { permanent_uuid: "93a1c25becc640a2b62bb5cc1cedfeaf" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 35377 } }
I20260928 21:54:03.830452 3831 leader_election.cc:288] T de68ddff5a9a41d0bd35a8c818ff7ba3 P 1819f1696bc14160929d3a11c851272a [CANDIDATE]: Term 1 election: Requested vote from peers 87b57a8ae47b4bf180b95361f946a3d2 (127.2.182.65:42055), 93a1c25becc640a2b62bb5cc1cedfeaf (127.2.182.66:35377)
I20260928 21:54:03.830646 3626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de68ddff5a9a41d0bd35a8c818ff7ba3" candidate_uuid: "1819f1696bc14160929d3a11c851272a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93a1c25becc640a2b62bb5cc1cedfeaf"
I20260928 21:54:03.830715 3626 raft_consensus.cc:3037] T de68ddff5a9a41d0bd35a8c818ff7ba3 P 93a1c25becc640a2b62bb5cc1cedfeaf [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:54:03.831540 3626 raft_consensus.cc:2454] T de68ddff5a9a41d0bd35a8c818ff7ba3 P 93a1c25becc640a2b62bb5cc1cedfeaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1819f1696bc14160929d3a11c851272a in term 1.
I20260928 21:54:03.831720 3492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de68ddff5a9a41d0bd35a8c818ff7ba3" candidate_uuid: "1819f1696bc14160929d3a11c851272a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87b57a8ae47b4bf180b95361f946a3d2"
I20260928 21:54:03.831773 3695 leader_election.cc:302] T de68ddff5a9a41d0bd35a8c818ff7ba3 P 1819f1696bc14160929d3a11c851272a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1819f1696bc14160929d3a11c851272a, 93a1c25becc640a2b62bb5cc1cedfeaf; no voters:
I20260928 21:54:03.831796 3492 raft_consensus.cc:3037] T de68ddff5a9a41d0bd35a8c818ff7ba3 P 87b57a8ae47b4bf180b95361f946a3d2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:54:03.831857 3831 raft_consensus.cc:2781] T de68ddff5a9a41d0bd35a8c818ff7ba3 P 1819f1696bc14160929d3a11c851272a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:54:03.832024 3831 raft_consensus.cc:686] T de68ddff5a9a41d0bd35a8c818ff7ba3 P 1819f1696bc14160929d3a11c851272a [term 1 LEADER]: Becoming Leader. State: Replica: 1819f1696bc14160929d3a11c851272a, State: Running, Role: LEADER
I20260928 21:54:03.832120 3831 consensus_queue.cc:237] T de68ddff5a9a41d0bd35a8c818ff7ba3 P 1819f1696bc14160929d3a11c851272a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87b57a8ae47b4bf180b95361f946a3d2" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 42055 } } peers { permanent_uuid: "1819f1696bc14160929d3a11c851272a" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 39045 } } peers { permanent_uuid: "93a1c25becc640a2b62bb5cc1cedfeaf" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 35377 } }
I20260928 21:54:03.832634 3492 raft_consensus.cc:2454] T de68ddff5a9a41d0bd35a8c818ff7ba3 P 87b57a8ae47b4bf180b95361f946a3d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1819f1696bc14160929d3a11c851272a in term 1.
I20260928 21:54:03.833297 3346 catalog_manager.cc:5495] T de68ddff5a9a41d0bd35a8c818ff7ba3 P 1819f1696bc14160929d3a11c851272a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1819f1696bc14160929d3a11c851272a (127.2.182.67). New cstate: current_term: 1 leader_uuid: "1819f1696bc14160929d3a11c851272a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87b57a8ae47b4bf180b95361f946a3d2" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 42055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1819f1696bc14160929d3a11c851272a" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 39045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93a1c25becc640a2b62bb5cc1cedfeaf" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 35377 } health_report { overall_health: UNKNOWN } } }
Sep 28 21:54:03 dist-test-slave-k6lz krb5kdc[3302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.182.65@KRBTEST.COM
W20260928 21:54:03.869277 3539 tablet.cc:2369] T de68ddff5a9a41d0bd35a8c818ff7ba3 P 87b57a8ae47b4bf180b95361f946a3d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:54:03.872179 3819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:03.863653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59446 (local address 127.2.182.65:42055)
0928 21:54:03.863714 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:03.863716 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:03.863966 (+ 250us) server_negotiation.cc:407] Connection header received
0928 21:54:03.863996 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:03.863997 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:03.864007 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:03.864057 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:03.864707 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:03.865191 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:03.865959 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:03.866168 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:03.866232 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:03.868852 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:03.868863 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:03.868864 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:03.868871 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:03.870905 (+ 2034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:03.871305 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:03.871309 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:03.871309 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:03.871409 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:03.871621 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:03.871622 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:03.871622 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:03.871777 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:03.871912 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:03.872018 (+ 106us) server_negotiation.cc:299] Negotiation successful
0928 21:54:03.872097 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
Sep 28 21:54:03 dist-test-slave-k6lz krb5kdc[3302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632443, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 21:54:03 dist-test-slave-k6lz krb5kdc[3302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632443, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.182.65@KRBTEST.COM
I20260928 21:54:03.896844 3819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:03.888812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59456 (local address 127.2.182.65:42055)
0928 21:54:03.888887 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:03.888889 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:03.888988 (+ 99us) server_negotiation.cc:407] Connection header received
0928 21:54:03.889043 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:03.889044 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:03.889054 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:03.889104 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:03.889715 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:03.890248 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:03.891065 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:03.891364 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:03.891436 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:03.893771 (+ 2335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:03.893772 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:03.893773 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:03.893781 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:03.895614 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:03.895964 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:03.895965 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:03.895966 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:03.896020 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:03.896236 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:03.896236 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:03.896236 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:03.896335 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:03.896407 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:03.896621 (+ 214us) server_negotiation.cc:299] Negotiation successful
0928 21:54:03.896755 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Sep 28 21:54:03 dist-test-slave-k6lz krb5kdc[3302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 21:54:03 dist-test-slave-k6lz krb5kdc[3302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.182.65@KRBTEST.COM
I20260928 21:54:03.920509 3819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:03.913063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59458 (local address 127.2.182.65:42055)
0928 21:54:03.913122 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:03.913125 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:03.913258 (+ 133us) server_negotiation.cc:407] Connection header received
0928 21:54:03.913290 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:03.913291 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:03.913302 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:03.913357 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:03.913936 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:03.914430 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:03.915218 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:03.915448 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:03.915505 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:03.917895 (+ 2390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:03.917897 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:03.917897 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:03.917905 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:03.919308 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:03.919714 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:03.919715 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:03.919716 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:03.919829 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:03.920002 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:03.920002 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:03.920003 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:03.920103 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:03.920183 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:03.920341 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 21:54:03.920417 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260928 21:54:03.920975 3451 tablet_service.cc:2908] Rejecting scan request for tablet de68ddff5a9a41d0bd35a8c818ff7ba3: Uninitialized: clean time has not yet been initialized
I20260928 21:54:03.922009 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 3409
I20260928 21:54:03.926745 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 3543
I20260928 21:54:03.930668 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 3677
I20260928 21:54:03.934576 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 3315
2026-09-28T21:54:03Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2015 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:54:03 dist-test-slave-k6lz krb5kdc[3847](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:54:03 dist-test-slave-k6lz krb5kdc[3847](info): set up 2 sockets
Sep 28 21:54:03 dist-test-slave-k6lz krb5kdc[3847](info): commencing operation
krb5kdc: starting...
W20260928 21:54:04.907023 2777 mini_kdc.cc:121] Time spent starting KDC: real 0.952s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Sep 28 21:54:04 dist-test-slave-k6lz krb5kdc[3847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:54:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:54:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab.
Entry for principal kudu/127.2.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab.
I20260928 21:54:05.031452 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:34929
--webserver_interface=127.2.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab
--principal=kudu/127.2.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:43373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.126:34929
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:54:05.112694 3860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:54:05.112908 3860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:54:05.112931 3860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:54:05.114176 3860 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:54:05.114230 3860 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 21:54:05.114243 3860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:54:05.114255 3860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:54:05.114265 3860 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:54:05.114276 3860 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:54:05.115913 3860 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:43373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.126:34929
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab
--principal=kudu/127.2.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:34929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:54:05.116151 3860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:54:05.116398 3860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:54:05.121630 3865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:05.121601 3868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:05.121620 3866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:05.122011 3860 server_base.cc:1034] running on GCE node
I20260928 21:54:05.122247 3860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:05.122543 3860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:05.123799 3860 hybrid_clock.cc:648] HybridClock initialized: now 1790632445123781 us; error 31 us; skew 500 ppm
Sep 28 21:54:05 dist-test-slave-k6lz krb5kdc[3847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632445, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:54:05.126755 3860 init.cc:377] Logged in from keytab as kudu/127.2.182.126@KRBTEST.COM (short username kudu)
I20260928 21:54:05.127226 3860 webserver.cc:460] Webserver started at http://127.2.182.126:38149/ using document root <none> and password file <none>
I20260928 21:54:05.127501 3860 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:05.127552 3860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:05.127969 3860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:54:05.129136 3860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/minicluster-data/master-0/data/instance:
uuid: "43159bea0e58402096e6d48ceadc44c1"
format_stamp: "Formatted at 2026-09-28 21:54:05 on dist-test-slave-k6lz"
server_key: "346df9475b5fb378ff57c2064ae80f1e"
server_key_iv: "f3467f75dc3acc0b69a7735fc4cfc630"
server_key_version: "encryptionkey@0"
I20260928 21:54:05.129477 3860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/minicluster-data/master-0/wal/instance:
uuid: "43159bea0e58402096e6d48ceadc44c1"
format_stamp: "Formatted at 2026-09-28 21:54:05 on dist-test-slave-k6lz"
server_key: "346df9475b5fb378ff57c2064ae80f1e"
server_key_iv: "f3467f75dc3acc0b69a7735fc4cfc630"
server_key_version: "encryptionkey@0"
I20260928 21:54:05.130983 3860 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:54:05.132279 3875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:05.132588 3860 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:54:05.132705 3860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/minicluster-data/master-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/minicluster-data/master-0/wal
uuid: "43159bea0e58402096e6d48ceadc44c1"
format_stamp: "Formatted at 2026-09-28 21:54:05 on dist-test-slave-k6lz"
server_key: "346df9475b5fb378ff57c2064ae80f1e"
server_key_iv: "f3467f75dc3acc0b69a7735fc4cfc630"
server_key_version: "encryptionkey@0"
I20260928 21:54:05.132803 3860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:05.145476 3860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:54:05.147941 3860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:54:05.148092 3860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:54:05.152660 3860 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.126:34929
I20260928 21:54:05.152797 3937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.126:34929 every 8 connection(s)
I20260928 21:54:05.153043 3860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
I20260928 21:54:05.153754 3938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:54:05.156190 3938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43159bea0e58402096e6d48ceadc44c1: Bootstrap starting.
I20260928 21:54:05.156312 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 3860
I20260928 21:54:05.156400 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/minicluster-data/master-0/wal/instance
I20260928 21:54:05.156556 2777 external_mini_cluster.cc:1428] Setting key 1e47d36d71759952d57de82c60c22534
I20260928 21:54:05.156821 3938 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 43159bea0e58402096e6d48ceadc44c1: Neither blocks nor log segments found. Creating new log.
I20260928 21:54:05.157135 3938 log.cc:824] T 00000000000000000000000000000000 P 43159bea0e58402096e6d48ceadc44c1: Log is configured to *not* fsync() on all Append() calls
I20260928 21:54:05.157853 3938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43159bea0e58402096e6d48ceadc44c1: No bootstrap required, opened a new log
I20260928 21:54:05.159610 3938 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43159bea0e58402096e6d48ceadc44c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43159bea0e58402096e6d48ceadc44c1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 34929 } }
I20260928 21:54:05.159760 3938 raft_consensus.cc:374] T 00000000000000000000000000000000 P 43159bea0e58402096e6d48ceadc44c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:54:05.159792 3938 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43159bea0e58402096e6d48ceadc44c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43159bea0e58402096e6d48ceadc44c1, State: Initialized, Role: FOLLOWER
I20260928 21:54:05.159919 3938 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43159bea0e58402096e6d48ceadc44c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43159bea0e58402096e6d48ceadc44c1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 34929 } }
I20260928 21:54:05.159992 3938 raft_consensus.cc:388] T 00000000000000000000000000000000 P 43159bea0e58402096e6d48ceadc44c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:54:05.160024 3938 raft_consensus.cc:482] T 00000000000000000000000000000000 P 43159bea0e58402096e6d48ceadc44c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:54:05.160071 3938 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43159bea0e58402096e6d48ceadc44c1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:54:05.160749 3938 raft_consensus.cc:504] T 00000000000000000000000000000000 P 43159bea0e58402096e6d48ceadc44c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43159bea0e58402096e6d48ceadc44c1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 34929 } }
I20260928 21:54:05.160871 3938 leader_election.cc:302] T 00000000000000000000000000000000 P 43159bea0e58402096e6d48ceadc44c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 43159bea0e58402096e6d48ceadc44c1; no voters:
I20260928 21:54:05.161041 3938 leader_election.cc:288] T 00000000000000000000000000000000 P 43159bea0e58402096e6d48ceadc44c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:54:05.161105 3943 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 43159bea0e58402096e6d48ceadc44c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:54:05.161310 3938 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43159bea0e58402096e6d48ceadc44c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:54:05.161299 3943 raft_consensus.cc:686] T 00000000000000000000000000000000 P 43159bea0e58402096e6d48ceadc44c1 [term 1 LEADER]: Becoming Leader. State: Replica: 43159bea0e58402096e6d48ceadc44c1, State: Running, Role: LEADER
I20260928 21:54:05.161510 3943 consensus_queue.cc:237] T 00000000000000000000000000000000 P 43159bea0e58402096e6d48ceadc44c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43159bea0e58402096e6d48ceadc44c1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 34929 } }
I20260928 21:54:05.161947 3945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43159bea0e58402096e6d48ceadc44c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43159bea0e58402096e6d48ceadc44c1. Latest consensus state: current_term: 1 leader_uuid: "43159bea0e58402096e6d48ceadc44c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43159bea0e58402096e6d48ceadc44c1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 34929 } } }
I20260928 21:54:05.162056 3945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43159bea0e58402096e6d48ceadc44c1 [sys.catalog]: This master's current role is: LEADER
I20260928 21:54:05.161940 3944 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43159bea0e58402096e6d48ceadc44c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "43159bea0e58402096e6d48ceadc44c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43159bea0e58402096e6d48ceadc44c1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 34929 } } }
I20260928 21:54:05.162328 3944 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43159bea0e58402096e6d48ceadc44c1 [sys.catalog]: This master's current role is: LEADER
I20260928 21:54:05.162695 3952 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:54:05.163196 3952 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 21:54:05 dist-test-slave-k6lz krb5kdc[3847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:54:05.167932 3952 catalog_manager.cc:1347] Generated new cluster ID: a3adb3013c554ebba0efd46a64266748
I20260928 21:54:05.167996 3952 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:54:05.168644 3941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:05.157175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54510 (local address 127.2.182.126:34929)
0928 21:54:05.157352 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:05.157360 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:05.157386 (+ 26us) server_negotiation.cc:407] Connection header received
0928 21:54:05.157808 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:05.157816 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:05.157862 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:05.157996 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:05.158737 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:05.160369 (+ 1632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:05.161392 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:05.161981 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:05.162095 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:05.164911 (+ 2816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:05.164915 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:05.164915 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:05.164927 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:05.166894 (+ 1967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:05.167250 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:05.167253 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:05.167254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:05.167400 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:05.167630 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:05.167630 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:05.167631 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:05.168020 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:05.168183 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:05.168292 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 21:54:05.168426 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":88,"threads_started":1}
I20260928 21:54:05.189287 3952 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:54:05.189975 3952 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:54:05.193979 3952 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 43159bea0e58402096e6d48ceadc44c1: Generated new TSK 0
I20260928 21:54:05.194176 3952 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.182.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab.
Entry for principal kudu/127.2.182.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab.
I20260928 21:54:05.223227 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.65:0
--local_ip_for_outbound_sockets=127.2.182.65
--webserver_interface=127.2.182.65
--webserver_port=0
--tserver_master_addrs=127.2.182.126:34929
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab
--principal=kudu/127.2.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:43373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:54:05.304658 3964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:54:05.304869 3964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:54:05.304893 3964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:54:05.306087 3964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:54:05.306131 3964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:54:05.306152 3964 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 21:54:05.306169 3964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.65
I20260928 21:54:05.307727 3964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:43373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab
--principal=kudu/127.2.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.182.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.182.126:34929
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.65
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:54:05.307960 3964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:54:05.308198 3964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:54:05.313376 3969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:05.313376 3970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:05.313462 3972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:05.313656 3964 server_base.cc:1034] running on GCE node
I20260928 21:54:05.313935 3964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:05.314167 3964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:05.315351 3964 hybrid_clock.cc:648] HybridClock initialized: now 1790632445315319 us; error 28 us; skew 500 ppm
Sep 28 21:54:05 dist-test-slave-k6lz krb5kdc[3847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632445, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:54:05.317970 3964 init.cc:377] Logged in from keytab as kudu/127.2.182.65@KRBTEST.COM (short username kudu)
I20260928 21:54:05.318414 3964 webserver.cc:460] Webserver started at http://127.2.182.65:39747/ using document root <none> and password file <none>
I20260928 21:54:05.318605 3964 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:05.318645 3964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:05.318733 3964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:54:05.319819 3964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/minicluster-data/ts-0/data/instance:
uuid: "59bb2db1feaa4da4b77ad2d68699feb1"
format_stamp: "Formatted at 2026-09-28 21:54:05 on dist-test-slave-k6lz"
server_key: "7c5eac4fc3b1321e32a5f76f386d6f72"
server_key_iv: "7a13a47de8b806a8e294a6bf08dd0900"
server_key_version: "encryptionkey@0"
I20260928 21:54:05.320112 3964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/minicluster-data/ts-0/wal/instance:
uuid: "59bb2db1feaa4da4b77ad2d68699feb1"
format_stamp: "Formatted at 2026-09-28 21:54:05 on dist-test-slave-k6lz"
server_key: "7c5eac4fc3b1321e32a5f76f386d6f72"
server_key_iv: "7a13a47de8b806a8e294a6bf08dd0900"
server_key_version: "encryptionkey@0"
I20260928 21:54:05.321326 3964 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:54:05.322022 3979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:05.322228 3964 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:54:05.322304 3964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/minicluster-data/ts-0/wal
uuid: "59bb2db1feaa4da4b77ad2d68699feb1"
format_stamp: "Formatted at 2026-09-28 21:54:05 on dist-test-slave-k6lz"
server_key: "7c5eac4fc3b1321e32a5f76f386d6f72"
server_key_iv: "7a13a47de8b806a8e294a6bf08dd0900"
server_key_version: "encryptionkey@0"
I20260928 21:54:05.322356 3964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:05.336086 3964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:54:05.338366 3964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:54:05.338475 3964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:54:05.339049 3964 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:54:05.339088 3964 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:05.339118 3964 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:54:05.339134 3964 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:05.345881 3964 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.65:41757
I20260928 21:54:05.346006 4094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.65:41757 every 8 connection(s)
I20260928 21:54:05.346293 3964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
Sep 28 21:54:05 dist-test-slave-k6lz krb5kdc[3847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632445, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.65@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:54:05.347961 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 3964
I20260928 21:54:05.348061 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/minicluster-data/ts-0/wal/instance
I20260928 21:54:05.348230 2777 external_mini_cluster.cc:1428] Setting key 56748665e99b1834188fdd4512474558
I20260928 21:54:05.350049 3941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:05.339468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.65:37979 (local address 127.2.182.126:34929)
0928 21:54:05.339532 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:05.339534 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:05.339798 (+ 264us) server_negotiation.cc:407] Connection header received
0928 21:54:05.340447 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:05.340448 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:05.340459 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:05.340515 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:05.341418 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:05.342223 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:05.343793 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:05.344130 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:05.344187 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:05.346960 (+ 2773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:05.346962 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:05.346963 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:05.346970 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:05.348451 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:05.349033 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:05.349034 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:05.349034 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:05.349100 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:05.349373 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:05.349373 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:05.349374 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:05.349517 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:05.349580 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:05.349862 (+ 282us) server_negotiation.cc:299] Negotiation successful
0928 21:54:05.349958 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 21:54:05.350135 3988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:05.339478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:34929 (local address 127.2.182.65:37979)
0928 21:54:05.339644 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:05.339693 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:05.340331 (+ 638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:05.340592 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:05.340596 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:05.340936 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:05.341250 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:05.341257 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:05.342337 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:05.342338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:05.343683 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:05.343686 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:05.344216 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:05.344217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:05.344369 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:05.344954 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:05.344965 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:05.346867 (+ 1902us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:05.348537 (+ 1670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:05.348544 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:05.348546 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:05.348957 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:05.349220 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:05.349220 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:05.349220 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:05.349320 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:05.349570 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:05.349574 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:05.349690 (+ 116us) client_negotiation.cc:769] Sending connection context
0928 21:54:05.349822 (+ 132us) client_negotiation.cc:240] Negotiation successful
0928 21:54:05.349956 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":86,"threads_started":1}
I20260928 21:54:05.350690 4095 heartbeater.cc:347] Connected to a master server at 127.2.182.126:34929
I20260928 21:54:05.350793 4095 heartbeater.cc:464] Registering TS with master...
I20260928 21:54:05.350998 4095 heartbeater.cc:511] Master 127.2.182.126:34929 requested a full tablet report, sending...
I20260928 21:54:05.351572 3891 ts_manager.cc:194] Registered new tserver with Master: 59bb2db1feaa4da4b77ad2d68699feb1 (127.2.182.65:41757)
I20260928 21:54:05.352447 3891 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.65@KRBTEST.COM'} at 127.2.182.65:37979
WARNING: no policy specified for kudu/127.2.182.66@KRBTEST.COM; defaulting to no policy
I20260928 21:54:05.358210 3941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:05.352700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.65:55129 (local address 127.2.182.126:34929)
0928 21:54:05.352740 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:05.352742 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:05.352750 (+ 8us) server_negotiation.cc:407] Connection header received
0928 21:54:05.352768 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:05.352769 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:05.352778 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:05.352836 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:05.353376 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:05.353863 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:05.354568 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:05.354772 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:05.354832 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:05.355777 (+ 945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:05.355778 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:05.355779 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:05.355786 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:05.357177 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:05.357504 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:05.357505 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:05.357506 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:05.357569 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:05.357725 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:05.357727 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:05.357727 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:05.357843 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:05.357897 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:05.358044 (+ 147us) server_negotiation.cc:299] Negotiation successful
0928 21:54:05.358116 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 21:54:05.358222 4102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:05.351659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:34929 (local address 127.2.182.65:55129)
0928 21:54:05.351845 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:05.351855 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:05.351918 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:05.352855 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:05.352857 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:05.353182 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:05.353278 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:05.353281 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:05.353950 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:05.353951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:05.354490 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:05.354491 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:05.354857 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:05.354858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:05.354927 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:05.355380 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:05.355384 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:05.355699 (+ 315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:05.357289 (+ 1590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:05.357290 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:05.357290 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:05.357442 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:05.357617 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:05.357618 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:05.357619 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:05.357677 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:05.357895 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:05.357896 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:05.357944 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 21:54:05.358013 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 21:54:05.358119 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"mutex_wait_us":2,"thread_start_us":74,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab.
Entry for principal kudu/127.2.182.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab.
I20260928 21:54:05.374861 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.66:0
--local_ip_for_outbound_sockets=127.2.182.66
--webserver_interface=127.2.182.66
--webserver_port=0
--tserver_master_addrs=127.2.182.126:34929
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab
--principal=kudu/127.2.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:43373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:54:05.455457 4104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:54:05.455672 4104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:54:05.455708 4104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:54:05.456916 4104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:54:05.456961 4104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:54:05.456990 4104 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 21:54:05.457012 4104 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.66
I20260928 21:54:05.458495 4104 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:43373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab
--principal=kudu/127.2.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.182.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.182.126:34929
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.66
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:54:05.458740 4104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:54:05.458954 4104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:54:05.464471 4104 server_base.cc:1034] running on GCE node
W20260928 21:54:05.464483 4112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:05.464566 4110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:05.464578 4109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:05.464905 4104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:05.465116 4104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:05.466275 4104 hybrid_clock.cc:648] HybridClock initialized: now 1790632445466264 us; error 27 us; skew 500 ppm
Sep 28 21:54:05 dist-test-slave-k6lz krb5kdc[3847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632445, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:54:05.468801 4104 init.cc:377] Logged in from keytab as kudu/127.2.182.66@KRBTEST.COM (short username kudu)
I20260928 21:54:05.469257 4104 webserver.cc:460] Webserver started at http://127.2.182.66:35343/ using document root <none> and password file <none>
I20260928 21:54:05.469475 4104 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:05.469517 4104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:05.469636 4104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:54:05.470665 4104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/minicluster-data/ts-1/data/instance:
uuid: "deebb36055154ca88052129c5c7ed3bc"
format_stamp: "Formatted at 2026-09-28 21:54:05 on dist-test-slave-k6lz"
server_key: "b508c9d8e7c120fff25b159f3aa019d6"
server_key_iv: "1607379e8e7c013ac346c5df81cf8b23"
server_key_version: "encryptionkey@0"
I20260928 21:54:05.470986 4104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/minicluster-data/ts-1/wal/instance:
uuid: "deebb36055154ca88052129c5c7ed3bc"
format_stamp: "Formatted at 2026-09-28 21:54:05 on dist-test-slave-k6lz"
server_key: "b508c9d8e7c120fff25b159f3aa019d6"
server_key_iv: "1607379e8e7c013ac346c5df81cf8b23"
server_key_version: "encryptionkey@0"
I20260928 21:54:05.472352 4104 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 21:54:05.473155 4119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:05.473304 4104 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 21:54:05.473388 4104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/minicluster-data/ts-1/wal
uuid: "deebb36055154ca88052129c5c7ed3bc"
format_stamp: "Formatted at 2026-09-28 21:54:05 on dist-test-slave-k6lz"
server_key: "b508c9d8e7c120fff25b159f3aa019d6"
server_key_iv: "1607379e8e7c013ac346c5df81cf8b23"
server_key_version: "encryptionkey@0"
I20260928 21:54:05.473449 4104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:05.490736 4104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:54:05.492995 4104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:54:05.493156 4104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:54:05.493777 4104 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:54:05.493832 4104 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:05.493862 4104 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:54:05.493878 4104 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:05.500885 4104 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.66:42025
I20260928 21:54:05.500975 4234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.66:42025 every 8 connection(s)
Sep 28 21:54:05 dist-test-slave-k6lz krb5kdc[3847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632445, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.66@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:54:05.501255 4104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
I20260928 21:54:05.504422 3941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:05.494305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.66:42329 (local address 127.2.182.126:34929)
0928 21:54:05.494367 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:05.494370 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:05.494512 (+ 142us) server_negotiation.cc:407] Connection header received
0928 21:54:05.495202 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:05.495203 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:05.495217 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:05.495276 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:05.496197 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:05.497011 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:05.498596 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:05.498895 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:05.498949 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:05.501744 (+ 2795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:05.501746 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:05.501747 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:05.501754 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:05.503097 (+ 1343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:05.503578 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:05.503579 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:05.503580 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:05.503640 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:05.503831 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:05.503832 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:05.503832 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:05.503948 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:05.503991 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:05.504287 (+ 296us) server_negotiation.cc:299] Negotiation successful
0928 21:54:05.504356 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 21:54:05.504504 4128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:05.494175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:34929 (local address 127.2.182.66:42329)
0928 21:54:05.494369 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:05.494419 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:05.495055 (+ 636us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:05.495396 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:05.495402 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:05.495772 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:05.496089 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:05.496096 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:05.497117 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:05.497118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:05.498485 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:05.498490 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:05.498972 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:05.498973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:05.499140 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:05.499797 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:05.499808 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:05.501631 (+ 1823us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:05.503201 (+ 1570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:05.503208 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:05.503210 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:05.503512 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:05.503691 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:05.503691 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:05.503691 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:05.503791 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:05.503990 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:05.503993 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:05.504118 (+ 125us) client_negotiation.cc:769] Sending connection context
0928 21:54:05.504203 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 21:54:05.504335 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":102,"threads_started":1}
I20260928 21:54:05.505096 4235 heartbeater.cc:347] Connected to a master server at 127.2.182.126:34929
I20260928 21:54:05.505180 4235 heartbeater.cc:464] Registering TS with master...
I20260928 21:54:05.505383 4235 heartbeater.cc:511] Master 127.2.182.126:34929 requested a full tablet report, sending...
I20260928 21:54:05.505895 3892 ts_manager.cc:194] Registered new tserver with Master: deebb36055154ca88052129c5c7ed3bc (127.2.182.66:42025)
I20260928 21:54:05.506409 3892 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.66@KRBTEST.COM'} at 127.2.182.66:42329
I20260928 21:54:05.510111 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 4104
I20260928 21:54:05.510188 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/minicluster-data/ts-1/wal/instance
I20260928 21:54:05.510352 2777 external_mini_cluster.cc:1428] Setting key 9f22e3f2cdeb0ad5d8713fb5108a33fc
I20260928 21:54:05.512847 3941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:05.506204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.66:37005 (local address 127.2.182.126:34929)
0928 21:54:05.506249 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:05.506251 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:05.506369 (+ 118us) server_negotiation.cc:407] Connection header received
0928 21:54:05.506420 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:05.506421 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:05.506432 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:05.506490 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:05.506907 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:05.507426 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:05.508242 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:05.508445 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:05.508508 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:05.509550 (+ 1042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:05.509551 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:05.509551 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:05.509560 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:05.511302 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:05.511769 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:05.511770 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:05.511771 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:05.511835 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:05.512084 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:05.512086 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:05.512086 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:05.512381 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:05.512464 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:05.512696 (+ 232us) server_negotiation.cc:299] Negotiation successful
0928 21:54:05.512768 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 21:54:05.513092 4241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:05.506144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:34929 (local address 127.2.182.66:37005)
0928 21:54:05.506302 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:05.506313 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:05.506377 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:05.506533 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:05.506534 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:05.506750 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:05.506814 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:05.506816 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:05.507529 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:05.507530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:05.508067 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:05.508125 (+ 58us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:05.508556 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:05.508557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:05.508628 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:05.509154 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:05.509159 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:05.509462 (+ 303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:05.511411 (+ 1949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:05.511413 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:05.511413 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:05.511697 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:05.511922 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:05.511923 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:05.511923 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:05.511989 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:05.512494 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:05.512495 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:05.512557 (+ 62us) client_negotiation.cc:769] Sending connection context
0928 21:54:05.512644 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 21:54:05.512907 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":77,"threads_started":1}
WARNING: no policy specified for kudu/127.2.182.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab.
Entry for principal kudu/127.2.182.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab.
I20260928 21:54:05.536513 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.67:0
--local_ip_for_outbound_sockets=127.2.182.67
--webserver_interface=127.2.182.67
--webserver_port=0
--tserver_master_addrs=127.2.182.126:34929
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab
--principal=kudu/127.2.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:43373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:54:05.618208 4244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:54:05.618424 4244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:54:05.618449 4244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:54:05.619673 4244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:54:05.619719 4244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:54:05.619740 4244 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 21:54:05.619756 4244 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.67
I20260928 21:54:05.621296 4244 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:43373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab
--principal=kudu/127.2.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.182.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.182.126:34929
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4244
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.67
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:54:05.621522 4244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:54:05.621721 4244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:54:05.626999 4249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:05.627045 4250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:05.627013 4252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:05.627172 4244 server_base.cc:1034] running on GCE node
I20260928 21:54:05.627547 4244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:05.627792 4244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:05.628949 4244 hybrid_clock.cc:648] HybridClock initialized: now 1790632445628918 us; error 49 us; skew 500 ppm
Sep 28 21:54:05 dist-test-slave-k6lz krb5kdc[3847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632445, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:54:05.631974 4244 init.cc:377] Logged in from keytab as kudu/127.2.182.67@KRBTEST.COM (short username kudu)
I20260928 21:54:05.632480 4244 webserver.cc:460] Webserver started at http://127.2.182.67:45039/ using document root <none> and password file <none>
I20260928 21:54:05.632704 4244 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:05.632750 4244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:05.632869 4244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:54:05.633936 4244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/minicluster-data/ts-2/data/instance:
uuid: "af52e7ef760c43fb8df9f26dc88ed828"
format_stamp: "Formatted at 2026-09-28 21:54:05 on dist-test-slave-k6lz"
server_key: "f06a8e99dacf5415791409b13a18de41"
server_key_iv: "6c98b6b764d0f30eca9e33cbd3ea94c0"
server_key_version: "encryptionkey@0"
I20260928 21:54:05.634253 4244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/minicluster-data/ts-2/wal/instance:
uuid: "af52e7ef760c43fb8df9f26dc88ed828"
format_stamp: "Formatted at 2026-09-28 21:54:05 on dist-test-slave-k6lz"
server_key: "f06a8e99dacf5415791409b13a18de41"
server_key_iv: "6c98b6b764d0f30eca9e33cbd3ea94c0"
server_key_version: "encryptionkey@0"
I20260928 21:54:05.635689 4244 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 21:54:05.636530 4259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:05.636803 4244 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:54:05.636884 4244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/minicluster-data/ts-2/wal
uuid: "af52e7ef760c43fb8df9f26dc88ed828"
format_stamp: "Formatted at 2026-09-28 21:54:05 on dist-test-slave-k6lz"
server_key: "f06a8e99dacf5415791409b13a18de41"
server_key_iv: "6c98b6b764d0f30eca9e33cbd3ea94c0"
server_key_version: "encryptionkey@0"
I20260928 21:54:05.636968 4244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:05.651290 4244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:54:05.653707 4244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:54:05.653847 4244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:54:05.654385 4244 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:54:05.654424 4244 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:05.654445 4244 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:54:05.654469 4244 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 21:54:05 dist-test-slave-k6lz krb5kdc[3847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632445, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.67@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:54:05.661239 4244 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.67:34801
I20260928 21:54:05.661311 4374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.67:34801 every 8 connection(s)
I20260928 21:54:05.661640 4244 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
I20260928 21:54:05.663820 3941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:05.654873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.67:59611 (local address 127.2.182.126:34929)
0928 21:54:05.654941 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:05.654943 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:05.655120 (+ 177us) server_negotiation.cc:407] Connection header received
0928 21:54:05.655654 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:05.655656 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:05.655669 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:05.655738 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:05.656395 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:05.656937 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:05.657992 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:05.658206 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:05.658263 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:05.660832 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:05.660834 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:05.660834 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:05.660842 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:05.662379 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:05.662877 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:05.662878 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:05.662878 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:05.662943 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:05.663181 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:05.663182 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:05.663182 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:05.663307 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:05.663380 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:05.663645 (+ 265us) server_negotiation.cc:299] Negotiation successful
0928 21:54:05.663729 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 21:54:05.663987 4268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:05.654821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:34929 (local address 127.2.182.67:59611)
0928 21:54:05.654994 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:05.655033 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:05.655562 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:05.655830 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:05.655833 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:05.656062 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:05.656296 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:05.656301 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:05.657022 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:05.657024 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:05.657918 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:05.657921 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:05.658285 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:05.658286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:05.658401 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:05.658890 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:05.658900 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:05.660736 (+ 1836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:05.662461 (+ 1725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:05.662468 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:05.662470 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:05.662815 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:05.663002 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:05.663003 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:05.663003 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:05.663119 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:05.663379 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:05.663385 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:05.663523 (+ 138us) client_negotiation.cc:769] Sending connection context
0928 21:54:05.663615 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 21:54:05.663775 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":99,"threads_started":1}
I20260928 21:54:05.664424 4375 heartbeater.cc:347] Connected to a master server at 127.2.182.126:34929
I20260928 21:54:05.664506 4375 heartbeater.cc:464] Registering TS with master...
I20260928 21:54:05.664711 4375 heartbeater.cc:511] Master 127.2.182.126:34929 requested a full tablet report, sending...
I20260928 21:54:05.665078 3891 ts_manager.cc:194] Registered new tserver with Master: af52e7ef760c43fb8df9f26dc88ed828 (127.2.182.67:34801)
I20260928 21:54:05.665519 3891 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.67@KRBTEST.COM'} at 127.2.182.67:59611
I20260928 21:54:05.670845 4381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:05.665421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:34929 (local address 127.2.182.67:47917)
0928 21:54:05.665536 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:05.665545 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:05.665601 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:05.665725 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:05.665725 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:05.665908 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:05.665970 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:05.665972 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:05.666607 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:05.666608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:05.667132 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:05.667134 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:05.667507 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:05.667509 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:05.667584 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:05.667970 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:05.667973 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:05.668306 (+ 333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:05.669861 (+ 1555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:05.669862 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:05.669862 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:05.670025 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:05.670228 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:05.670228 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:05.670229 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:05.670292 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:05.670503 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:05.670503 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:05.670571 (+ 68us) client_negotiation.cc:769] Sending connection context
0928 21:54:05.670627 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 21:54:05.670736 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":66,"threads_started":1}
I20260928 21:54:05.670836 3941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:05.665488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.67:47917 (local address 127.2.182.126:34929)
0928 21:54:05.665527 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:05.665529 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:05.665602 (+ 73us) server_negotiation.cc:407] Connection header received
0928 21:54:05.665657 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:05.665659 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:05.665669 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:05.665720 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:05.666042 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:05.666516 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:05.667205 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:05.667449 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:05.667504 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:05.668388 (+ 884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:05.668389 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:05.668390 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:05.668397 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:05.669717 (+ 1320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:05.670117 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:05.670119 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:05.670119 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:05.670177 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:05.670341 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:05.670341 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:05.670342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:05.670451 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:05.670505 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:05.670661 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 21:54:05.670732 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 21:54:05.671563 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 4244
I20260928 21:54:05.671630 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/minicluster-data/ts-2/wal/instance
I20260928 21:54:05.671770 2777 external_mini_cluster.cc:1428] Setting key da40a4b3f0e57e3f533e239b1032f46b
I20260928 21:54:05.672515 2777 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 21:54:05 dist-test-slave-k6lz krb5kdc[3847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632445, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 21:54:05 dist-test-slave-k6lz krb5kdc[3847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632445, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:54:05.703444 3941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:05.695657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54520 (local address 127.2.182.126:34929)
0928 21:54:05.695724 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:05.695726 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:05.695738 (+ 12us) server_negotiation.cc:407] Connection header received
0928 21:54:05.695820 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:05.695821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:05.695834 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:05.695937 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:05.696759 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:05.697331 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:05.698083 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:05.698272 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:05.698320 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:05.700771 (+ 2451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:05.700773 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:05.700774 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:05.700783 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:05.702446 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:05.702829 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:05.702830 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:05.702830 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:05.702892 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:05.703023 (+ 131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:05.703023 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:05.703023 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:05.703124 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:05.703195 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:05.703259 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 21:54:05.703319 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 21:54:05.704988 3891 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54520:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:54:05.706598 3891 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:54:05.713258 4396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:05.709107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.65:41757 (local address 127.0.0.1:56938)
0928 21:54:05.709399 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:05.709410 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:05.709459 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:05.709623 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:05.709626 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:05.709628 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:54:05.709696 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:05.709697 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:05.711177 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:05.711178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:05.712238 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:05.712241 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:05.712906 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:05.712907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:05.712997 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:05.713011 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 21:54:05.713074 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 21:54:05.713132 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":84,"threads_started":1}
I20260928 21:54:05.713245 4395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:05.709221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56938 (local address 127.2.182.65:41757)
0928 21:54:05.709431 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:05.709436 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:05.709450 (+ 14us) server_negotiation.cc:407] Connection header received
0928 21:54:05.709526 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:05.709529 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:05.709544 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:05.709617 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:54:05.709767 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:05.711070 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:05.712318 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:05.712823 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:05.712887 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:05.712949 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:05.713095 (+ 146us) server_negotiation.cc:299] Negotiation successful
0928 21:54:05.713133 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":105,"threads_started":1}
I20260928 21:54:05.713428 4394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:05.709206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56710 (local address 127.2.182.67:34801)
0928 21:54:05.709683 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:05.709687 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:05.709696 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:54:05.709727 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:05.709729 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:05.709743 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:05.709957 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:54:05.710015 (+ 58us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:05.710702 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:05.712077 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:05.712826 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:05.712904 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:05.712959 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:05.713299 (+ 340us) server_negotiation.cc:299] Negotiation successful
0928 21:54:05.713341 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":86,"threads_started":1}
I20260928 21:54:05.713611 4397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:05.709500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48384 (local address 127.2.182.66:42025)
0928 21:54:05.709652 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:05.709656 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:05.709667 (+ 11us) server_negotiation.cc:407] Connection header received
0928 21:54:05.709706 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:05.709707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:05.709722 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:05.709800 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:54:05.709980 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:05.711081 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:05.712354 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:05.713056 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:05.713113 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:05.713173 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:05.713465 (+ 292us) server_negotiation.cc:299] Negotiation successful
0928 21:54:05.713507 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":72,"threads_started":1}
I20260928 21:54:05.713397 4392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:05.709295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.66:42025 (local address 127.0.0.1:48384)
0928 21:54:05.709473 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:05.709480 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:05.709513 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:05.709848 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:05.709849 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:05.709851 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:54:05.709896 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:05.709897 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:05.711403 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:05.711404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:05.712289 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:05.712291 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:05.713205 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:05.713206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:05.713259 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:05.713263 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:54:05.713307 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 21:54:05.713339 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":67,"threads_started":1}
I20260928 21:54:05.713260 4393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:05.709107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.67:34801 (local address 127.0.0.1:56710)
0928 21:54:05.709270 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:05.709313 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:05.709379 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:05.709833 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:05.709834 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:05.709836 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:54:05.709888 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:05.709890 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:05.710782 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:05.710783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:05.711984 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:05.711986 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:05.712992 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:05.712994 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:05.713055 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:05.713060 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 21:54:05.713106 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 21:54:05.713150 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":58,"threads_started":1}
I20260928 21:54:05.714377 4169 tablet_service.cc:1467] Processing CreateTablet for tablet e86f14a768ca4a5a8f6d9dcbfa7efc12 (DEFAULT_TABLE table=test-table [id=53f080f99be8477987bbeadd636fdbf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:54:05.714385 4309 tablet_service.cc:1467] Processing CreateTablet for tablet e86f14a768ca4a5a8f6d9dcbfa7efc12 (DEFAULT_TABLE table=test-table [id=53f080f99be8477987bbeadd636fdbf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:54:05.714387 4029 tablet_service.cc:1467] Processing CreateTablet for tablet e86f14a768ca4a5a8f6d9dcbfa7efc12 (DEFAULT_TABLE table=test-table [id=53f080f99be8477987bbeadd636fdbf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:54:05.714701 4029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e86f14a768ca4a5a8f6d9dcbfa7efc12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:54:05.714702 4309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e86f14a768ca4a5a8f6d9dcbfa7efc12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:54:05.714708 4169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e86f14a768ca4a5a8f6d9dcbfa7efc12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:54:05.717691 4398 tablet_bootstrap.cc:492] T e86f14a768ca4a5a8f6d9dcbfa7efc12 P deebb36055154ca88052129c5c7ed3bc: Bootstrap starting.
I20260928 21:54:05.718174 4400 tablet_bootstrap.cc:492] T e86f14a768ca4a5a8f6d9dcbfa7efc12 P 59bb2db1feaa4da4b77ad2d68699feb1: Bootstrap starting.
I20260928 21:54:05.718245 4399 tablet_bootstrap.cc:492] T e86f14a768ca4a5a8f6d9dcbfa7efc12 P af52e7ef760c43fb8df9f26dc88ed828: Bootstrap starting.
I20260928 21:54:05.718699 4398 tablet_bootstrap.cc:654] T e86f14a768ca4a5a8f6d9dcbfa7efc12 P deebb36055154ca88052129c5c7ed3bc: Neither blocks nor log segments found. Creating new log.
I20260928 21:54:05.718813 4400 tablet_bootstrap.cc:654] T e86f14a768ca4a5a8f6d9dcbfa7efc12 P 59bb2db1feaa4da4b77ad2d68699feb1: Neither blocks nor log segments found. Creating new log.
I20260928 21:54:05.719007 4398 log.cc:824] T e86f14a768ca4a5a8f6d9dcbfa7efc12 P deebb36055154ca88052129c5c7ed3bc: Log is configured to *not* fsync() on all Append() calls
I20260928 21:54:05.719081 4399 tablet_bootstrap.cc:654] T e86f14a768ca4a5a8f6d9dcbfa7efc12 P af52e7ef760c43fb8df9f26dc88ed828: Neither blocks nor log segments found. Creating new log.
I20260928 21:54:05.719118 4400 log.cc:824] T e86f14a768ca4a5a8f6d9dcbfa7efc12 P 59bb2db1feaa4da4b77ad2d68699feb1: Log is configured to *not* fsync() on all Append() calls
I20260928 21:54:05.719424 4399 log.cc:824] T e86f14a768ca4a5a8f6d9dcbfa7efc12 P af52e7ef760c43fb8df9f26dc88ed828: Log is configured to *not* fsync() on all Append() calls
I20260928 21:54:05.719826 4398 tablet_bootstrap.cc:492] T e86f14a768ca4a5a8f6d9dcbfa7efc12 P deebb36055154ca88052129c5c7ed3bc: No bootstrap required, opened a new log
I20260928 21:54:05.719879 4398 ts_tablet_manager.cc:1397] T e86f14a768ca4a5a8f6d9dcbfa7efc12 P deebb36055154ca88052129c5c7ed3bc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:54:05.720039 4399 tablet_bootstrap.cc:492] T e86f14a768ca4a5a8f6d9dcbfa7efc12 P af52e7ef760c43fb8df9f26dc88ed828: No bootstrap required, opened a new log
I20260928 21:54:05.720094 4399 ts_tablet_manager.cc:1397] T e86f14a768ca4a5a8f6d9dcbfa7efc12 P af52e7ef760c43fb8df9f26dc88ed828: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:54:05.721285 4398 raft_consensus.cc:348] T e86f14a768ca4a5a8f6d9dcbfa7efc12 P deebb36055154ca88052129c5c7ed3bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59bb2db1feaa4da4b77ad2d68699feb1" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 41757 } } peers { permanent_uuid: "af52e7ef760c43fb8df9f26dc88ed828" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 34801 } } peers { permanent_uuid: "deebb36055154ca88052129c5c7ed3bc" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 42025 } }
I20260928 21:54:05.721417 4398 raft_consensus.cc:374] T e86f14a768ca4a5a8f6d9dcbfa7efc12 P deebb36055154ca88052129c5c7ed3bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:54:05.721439 4398 raft_consensus.cc:729] T e86f14a768ca4a5a8f6d9dcbfa7efc12 P deebb36055154ca88052129c5c7ed3bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: deebb36055154ca88052129c5c7ed3bc, State: Initialized, Role: FOLLOWER
I20260928 21:54:05.721400 4399 raft_consensus.cc:348] T e86f14a768ca4a5a8f6d9dcbfa7efc12 P af52e7ef760c43fb8df9f26dc88ed828 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59bb2db1feaa4da4b77ad2d68699feb1" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 41757 } } peers { permanent_uuid: "af52e7ef760c43fb8df9f26dc88ed828" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 34801 } } peers { permanent_uuid: "deebb36055154ca88052129c5c7ed3bc" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 42025 } }
I20260928 21:54:05.721545 4399 raft_consensus.cc:374] T e86f14a768ca4a5a8f6d9dcbfa7efc12 P af52e7ef760c43fb8df9f26dc88ed828 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:54:05.721539 4398 consensus_queue.cc:260] T e86f14a768ca4a5a8f6d9dcbfa7efc12 P deebb36055154ca88052129c5c7ed3bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59bb2db1feaa4da4b77ad2d68699feb1" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 41757 } } peers { permanent_uuid: "af52e7ef760c43fb8df9f26dc88ed828" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 34801 } } peers { permanent_uuid: "deebb36055154ca88052129c5c7ed3bc" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 42025 } }
I20260928 21:54:05.721586 4399 raft_consensus.cc:729] T e86f14a768ca4a5a8f6d9dcbfa7efc12 P af52e7ef760c43fb8df9f26dc88ed828 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af52e7ef760c43fb8df9f26dc88ed828, State: Initialized, Role: FOLLOWER
I20260928 21:54:05.721690 4399 consensus_queue.cc:260] T e86f14a768ca4a5a8f6d9dcbfa7efc12 P af52e7ef760c43fb8df9f26dc88ed828 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59bb2db1feaa4da4b77ad2d68699feb1" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 41757 } } peers { permanent_uuid: "af52e7ef760c43fb8df9f26dc88ed828" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 34801 } } peers { permanent_uuid: "deebb36055154ca88052129c5c7ed3bc" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 42025 } }
I20260928 21:54:05.721850 4235 heartbeater.cc:503] Master 127.2.182.126:34929 was elected leader, sending a full tablet report...
I20260928 21:54:05.721948 4399 ts_tablet_manager.cc:1428] T e86f14a768ca4a5a8f6d9dcbfa7efc12 P af52e7ef760c43fb8df9f26dc88ed828: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 21:54:05.722067 4375 heartbeater.cc:503] Master 127.2.182.126:34929 was elected leader, sending a full tablet report...
I20260928 21:54:05.722177 4398 ts_tablet_manager.cc:1428] T e86f14a768ca4a5a8f6d9dcbfa7efc12 P deebb36055154ca88052129c5c7ed3bc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 21:54:05.722596 4400 tablet_bootstrap.cc:492] T e86f14a768ca4a5a8f6d9dcbfa7efc12 P 59bb2db1feaa4da4b77ad2d68699feb1: No bootstrap required, opened a new log
I20260928 21:54:05.722681 4400 ts_tablet_manager.cc:1397] T e86f14a768ca4a5a8f6d9dcbfa7efc12 P 59bb2db1feaa4da4b77ad2d68699feb1: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260928 21:54:05.724300 4400 raft_consensus.cc:348] T e86f14a768ca4a5a8f6d9dcbfa7efc12 P 59bb2db1feaa4da4b77ad2d68699feb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59bb2db1feaa4da4b77ad2d68699feb1" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 41757 } } peers { permanent_uuid: "af52e7ef760c43fb8df9f26dc88ed828" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 34801 } } peers { permanent_uuid: "deebb36055154ca88052129c5c7ed3bc" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 42025 } }
I20260928 21:54:05.724460 4400 raft_consensus.cc:374] T e86f14a768ca4a5a8f6d9dcbfa7efc12 P 59bb2db1feaa4da4b77ad2d68699feb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:54:05.724485 4400 raft_consensus.cc:729] T e86f14a768ca4a5a8f6d9dcbfa7efc12 P 59bb2db1feaa4da4b77ad2d68699feb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59bb2db1feaa4da4b77ad2d68699feb1, State: Initialized, Role: FOLLOWER
I20260928 21:54:05.724594 4400 consensus_queue.cc:260] T e86f14a768ca4a5a8f6d9dcbfa7efc12 P 59bb2db1feaa4da4b77ad2d68699feb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59bb2db1feaa4da4b77ad2d68699feb1" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 41757 } } peers { permanent_uuid: "af52e7ef760c43fb8df9f26dc88ed828" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 34801 } } peers { permanent_uuid: "deebb36055154ca88052129c5c7ed3bc" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 42025 } }
I20260928 21:54:05.724838 4400 ts_tablet_manager.cc:1428] T e86f14a768ca4a5a8f6d9dcbfa7efc12 P 59bb2db1feaa4da4b77ad2d68699feb1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 21:54:05.724869 4095 heartbeater.cc:503] Master 127.2.182.126:34929 was elected leader, sending a full tablet report...
W20260928 21:54:05.751907 4236 tablet.cc:2369] T e86f14a768ca4a5a8f6d9dcbfa7efc12 P deebb36055154ca88052129c5c7ed3bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:54:05.797305 4405 raft_consensus.cc:482] T e86f14a768ca4a5a8f6d9dcbfa7efc12 P af52e7ef760c43fb8df9f26dc88ed828 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:54:05.797430 4405 raft_consensus.cc:504] T e86f14a768ca4a5a8f6d9dcbfa7efc12 P af52e7ef760c43fb8df9f26dc88ed828 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59bb2db1feaa4da4b77ad2d68699feb1" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 41757 } } peers { permanent_uuid: "af52e7ef760c43fb8df9f26dc88ed828" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 34801 } } peers { permanent_uuid: "deebb36055154ca88052129c5c7ed3bc" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 42025 } }
I20260928 21:54:05.797772 4405 leader_election.cc:288] T e86f14a768ca4a5a8f6d9dcbfa7efc12 P af52e7ef760c43fb8df9f26dc88ed828 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59bb2db1feaa4da4b77ad2d68699feb1 (127.2.182.65:41757), deebb36055154ca88052129c5c7ed3bc (127.2.182.66:42025)
I20260928 21:54:05.801789 4395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:05.798105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.67:53211 (local address 127.2.182.65:41757)
0928 21:54:05.798208 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:05.798210 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:05.798309 (+ 99us) server_negotiation.cc:407] Connection header received
0928 21:54:05.798349 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:05.798350 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:05.798358 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:05.798405 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:54:05.798550 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:05.799317 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:05.800834 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:05.801416 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:05.801482 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:05.801491 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:05.801672 (+ 181us) server_negotiation.cc:299] Negotiation successful
0928 21:54:05.801714 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 21:54:05.801789 4397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:05.798206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.67:41159 (local address 127.2.182.66:42025)
0928 21:54:05.798250 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:05.798251 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:05.798258 (+ 7us) server_negotiation.cc:407] Connection header received
0928 21:54:05.798282 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:05.798283 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:05.798292 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:05.798345 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:54:05.798540 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:05.799374 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:05.800835 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:05.801404 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:05.801482 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:05.801491 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:05.801672 (+ 181us) server_negotiation.cc:299] Negotiation successful
0928 21:54:05.801713 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 21:54:05.801805 4268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:05.798012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.66:42025 (local address 127.2.182.67:41159)
0928 21:54:05.798100 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:05.798113 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:05.798178 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:05.798378 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:05.798379 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:05.798380 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:54:05.798454 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:05.798456 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:05.799453 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:05.799454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:05.800718 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:05.800722 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:05.801498 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:05.801499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:05.801600 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:05.801602 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 21:54:05.801660 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 21:54:05.801716 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"mutex_wait_us":65}
I20260928 21:54:05.802063 4407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:05.798012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.65:41757 (local address 127.2.182.67:53211)
0928 21:54:05.798235 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:05.798246 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:05.798301 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:05.798424 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:05.798424 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:05.798438 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:54:05.798488 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:05.798490 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:05.799426 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:05.799427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:05.800718 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:05.800722 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:05.801498 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:05.801499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:05.801596 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:05.801601 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 21:54:05.801660 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 21:54:05.801716 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"mutex_wait_us":35,"thread_start_us":120,"threads_started":1}
I20260928 21:54:05.802251 4189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e86f14a768ca4a5a8f6d9dcbfa7efc12" candidate_uuid: "af52e7ef760c43fb8df9f26dc88ed828" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "deebb36055154ca88052129c5c7ed3bc" is_pre_election: true
I20260928 21:54:05.802417 4189 raft_consensus.cc:2454] T e86f14a768ca4a5a8f6d9dcbfa7efc12 P deebb36055154ca88052129c5c7ed3bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af52e7ef760c43fb8df9f26dc88ed828 in term 0.
I20260928 21:54:05.802381 4049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e86f14a768ca4a5a8f6d9dcbfa7efc12" candidate_uuid: "af52e7ef760c43fb8df9f26dc88ed828" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59bb2db1feaa4da4b77ad2d68699feb1" is_pre_election: true
I20260928 21:54:05.802518 4049 raft_consensus.cc:2454] T e86f14a768ca4a5a8f6d9dcbfa7efc12 P 59bb2db1feaa4da4b77ad2d68699feb1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af52e7ef760c43fb8df9f26dc88ed828 in term 0.
I20260928 21:54:05.802667 4263 leader_election.cc:302] T e86f14a768ca4a5a8f6d9dcbfa7efc12 P af52e7ef760c43fb8df9f26dc88ed828 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af52e7ef760c43fb8df9f26dc88ed828, deebb36055154ca88052129c5c7ed3bc; no voters:
I20260928 21:54:05.802819 4405 raft_consensus.cc:2781] T e86f14a768ca4a5a8f6d9dcbfa7efc12 P af52e7ef760c43fb8df9f26dc88ed828 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:54:05.802923 4405 raft_consensus.cc:482] T e86f14a768ca4a5a8f6d9dcbfa7efc12 P af52e7ef760c43fb8df9f26dc88ed828 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:54:05.802945 4405 raft_consensus.cc:3037] T e86f14a768ca4a5a8f6d9dcbfa7efc12 P af52e7ef760c43fb8df9f26dc88ed828 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:54:05.803689 4405 raft_consensus.cc:504] T e86f14a768ca4a5a8f6d9dcbfa7efc12 P af52e7ef760c43fb8df9f26dc88ed828 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59bb2db1feaa4da4b77ad2d68699feb1" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 41757 } } peers { permanent_uuid: "af52e7ef760c43fb8df9f26dc88ed828" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 34801 } } peers { permanent_uuid: "deebb36055154ca88052129c5c7ed3bc" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 42025 } }
I20260928 21:54:05.803809 4405 leader_election.cc:288] T e86f14a768ca4a5a8f6d9dcbfa7efc12 P af52e7ef760c43fb8df9f26dc88ed828 [CANDIDATE]: Term 1 election: Requested vote from peers 59bb2db1feaa4da4b77ad2d68699feb1 (127.2.182.65:41757), deebb36055154ca88052129c5c7ed3bc (127.2.182.66:42025)
I20260928 21:54:05.803963 4189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e86f14a768ca4a5a8f6d9dcbfa7efc12" candidate_uuid: "af52e7ef760c43fb8df9f26dc88ed828" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "deebb36055154ca88052129c5c7ed3bc"
I20260928 21:54:05.803963 4049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e86f14a768ca4a5a8f6d9dcbfa7efc12" candidate_uuid: "af52e7ef760c43fb8df9f26dc88ed828" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59bb2db1feaa4da4b77ad2d68699feb1"
I20260928 21:54:05.804047 4189 raft_consensus.cc:3037] T e86f14a768ca4a5a8f6d9dcbfa7efc12 P deebb36055154ca88052129c5c7ed3bc [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:54:05.804047 4049 raft_consensus.cc:3037] T e86f14a768ca4a5a8f6d9dcbfa7efc12 P 59bb2db1feaa4da4b77ad2d68699feb1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:54:05.804886 4189 raft_consensus.cc:2454] T e86f14a768ca4a5a8f6d9dcbfa7efc12 P deebb36055154ca88052129c5c7ed3bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af52e7ef760c43fb8df9f26dc88ed828 in term 1.
I20260928 21:54:05.804886 4049 raft_consensus.cc:2454] T e86f14a768ca4a5a8f6d9dcbfa7efc12 P 59bb2db1feaa4da4b77ad2d68699feb1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af52e7ef760c43fb8df9f26dc88ed828 in term 1.
I20260928 21:54:05.805058 4263 leader_election.cc:302] T e86f14a768ca4a5a8f6d9dcbfa7efc12 P af52e7ef760c43fb8df9f26dc88ed828 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af52e7ef760c43fb8df9f26dc88ed828, deebb36055154ca88052129c5c7ed3bc; no voters:
I20260928 21:54:05.805169 4405 raft_consensus.cc:2781] T e86f14a768ca4a5a8f6d9dcbfa7efc12 P af52e7ef760c43fb8df9f26dc88ed828 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:54:05.805311 4405 raft_consensus.cc:686] T e86f14a768ca4a5a8f6d9dcbfa7efc12 P af52e7ef760c43fb8df9f26dc88ed828 [term 1 LEADER]: Becoming Leader. State: Replica: af52e7ef760c43fb8df9f26dc88ed828, State: Running, Role: LEADER
I20260928 21:54:05.805421 4405 consensus_queue.cc:237] T e86f14a768ca4a5a8f6d9dcbfa7efc12 P af52e7ef760c43fb8df9f26dc88ed828 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59bb2db1feaa4da4b77ad2d68699feb1" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 41757 } } peers { permanent_uuid: "af52e7ef760c43fb8df9f26dc88ed828" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 34801 } } peers { permanent_uuid: "deebb36055154ca88052129c5c7ed3bc" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 42025 } }
I20260928 21:54:05.806273 3891 catalog_manager.cc:5495] T e86f14a768ca4a5a8f6d9dcbfa7efc12 P af52e7ef760c43fb8df9f26dc88ed828 reported cstate change: term changed from 0 to 1, leader changed from <none> to af52e7ef760c43fb8df9f26dc88ed828 (127.2.182.67). New cstate: current_term: 1 leader_uuid: "af52e7ef760c43fb8df9f26dc88ed828" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59bb2db1feaa4da4b77ad2d68699feb1" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 41757 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af52e7ef760c43fb8df9f26dc88ed828" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 34801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "deebb36055154ca88052129c5c7ed3bc" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 42025 } health_report { overall_health: UNKNOWN } } }
I20260928 21:54:05.831236 4394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:05.828741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56722 (local address 127.2.182.67:34801)
0928 21:54:05.828792 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:05.828794 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:05.828802 (+ 8us) server_negotiation.cc:407] Connection header received
0928 21:54:05.828841 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:05.828842 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:05.828861 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:05.828914 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:54:05.829042 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:05.829522 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:05.830416 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:05.830634 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:05.830698 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:05.830904 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:54:05.831002 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:54:05.831071 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:05.831113 (+ 42us) server_negotiation.cc:299] Negotiation successful
0928 21:54:05.831138 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 21:54:05.833060 4049 raft_consensus.cc:1261] T e86f14a768ca4a5a8f6d9dcbfa7efc12 P 59bb2db1feaa4da4b77ad2d68699feb1 [term 1 FOLLOWER]: Refusing update from remote peer af52e7ef760c43fb8df9f26dc88ed828: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:54:05.833086 4189 raft_consensus.cc:1261] T e86f14a768ca4a5a8f6d9dcbfa7efc12 P deebb36055154ca88052129c5c7ed3bc [term 1 FOLLOWER]: Refusing update from remote peer af52e7ef760c43fb8df9f26dc88ed828: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:54:05.833424 4405 consensus_queue.cc:1035] T e86f14a768ca4a5a8f6d9dcbfa7efc12 P af52e7ef760c43fb8df9f26dc88ed828 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59bb2db1feaa4da4b77ad2d68699feb1" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 41757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:54:05.833587 4408 consensus_queue.cc:1035] T e86f14a768ca4a5a8f6d9dcbfa7efc12 P af52e7ef760c43fb8df9f26dc88ed828 [LEADER]: Connected to new peer: Peer: permanent_uuid: "deebb36055154ca88052129c5c7ed3bc" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 42025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:54:05.835783 4411 mvcc.cc:205] Tried to move back new op lower bound from 7334430498129408000 to 7334430498019614720. Current Snapshot: MvccSnapshot[applied={T|T < 7334430498129408000}]
I20260928 21:54:05.836608 4413 mvcc.cc:205] Tried to move back new op lower bound from 7334430498129408000 to 7334430498019614720. Current Snapshot: MvccSnapshot[applied={T|T < 7334430498129408000 or (T in {7334430498129408000})}]
I20260928 21:54:05.839632 3891 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54520:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:54:05.839725 3891 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54520:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:54:05.840533 3891 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 43159bea0e58402096e6d48ceadc44c1: Sending DeleteTablet for 3 replicas of tablet e86f14a768ca4a5a8f6d9dcbfa7efc12
I20260928 21:54:05.840885 4309 tablet_service.cc:1514] Processing DeleteTablet for tablet e86f14a768ca4a5a8f6d9dcbfa7efc12 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:54:05 UTC) from {username='kudu', principal='kudu/127.2.182.126@KRBTEST.COM'} at 127.0.0.1:56710
I20260928 21:54:05.840991 4169 tablet_service.cc:1514] Processing DeleteTablet for tablet e86f14a768ca4a5a8f6d9dcbfa7efc12 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:54:05 UTC) from {username='kudu', principal='kudu/127.2.182.126@KRBTEST.COM'} at 127.0.0.1:48384
I20260928 21:54:05.840991 4029 tablet_service.cc:1514] Processing DeleteTablet for tablet e86f14a768ca4a5a8f6d9dcbfa7efc12 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:54:05 UTC) from {username='kudu', principal='kudu/127.2.182.126@KRBTEST.COM'} at 127.0.0.1:56938
I20260928 21:54:05.841213 4421 tablet_replica.cc:330] T e86f14a768ca4a5a8f6d9dcbfa7efc12 P 59bb2db1feaa4da4b77ad2d68699feb1: stopping tablet replica
I20260928 21:54:05.841346 4421 raft_consensus.cc:2229] T e86f14a768ca4a5a8f6d9dcbfa7efc12 P 59bb2db1feaa4da4b77ad2d68699feb1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:54:05.841456 4421 raft_consensus.cc:2258] T e86f14a768ca4a5a8f6d9dcbfa7efc12 P 59bb2db1feaa4da4b77ad2d68699feb1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:54:05.841454 3891 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54520:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:54:05.841606 3891 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:54:05.841836 4422 tablet_replica.cc:330] T e86f14a768ca4a5a8f6d9dcbfa7efc12 P deebb36055154ca88052129c5c7ed3bc: stopping tablet replica
I20260928 21:54:05.841890 4420 tablet_replica.cc:330] T e86f14a768ca4a5a8f6d9dcbfa7efc12 P af52e7ef760c43fb8df9f26dc88ed828: stopping tablet replica
I20260928 21:54:05.841982 4420 raft_consensus.cc:2229] T e86f14a768ca4a5a8f6d9dcbfa7efc12 P af52e7ef760c43fb8df9f26dc88ed828 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:54:05.842097 4420 raft_consensus.cc:2258] T e86f14a768ca4a5a8f6d9dcbfa7efc12 P af52e7ef760c43fb8df9f26dc88ed828 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:54:05.841935 4422 raft_consensus.cc:2229] T e86f14a768ca4a5a8f6d9dcbfa7efc12 P deebb36055154ca88052129c5c7ed3bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:54:05.842455 4422 raft_consensus.cc:2258] T e86f14a768ca4a5a8f6d9dcbfa7efc12 P deebb36055154ca88052129c5c7ed3bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:54:05.842780 4421 ts_tablet_manager.cc:1905] T e86f14a768ca4a5a8f6d9dcbfa7efc12 P 59bb2db1feaa4da4b77ad2d68699feb1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:54:05.842919 4029 tablet_service.cc:1467] Processing CreateTablet for tablet 7dcb2ce5ec9d46b48fd8a736e24897f0 (DEFAULT_TABLE table=test-table [id=07aa8cf91ec64877b634715622e88096]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:54:05.842958 4169 tablet_service.cc:1467] Processing CreateTablet for tablet 7dcb2ce5ec9d46b48fd8a736e24897f0 (DEFAULT_TABLE table=test-table [id=07aa8cf91ec64877b634715622e88096]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:54:05.842993 4309 tablet_service.cc:1467] Processing CreateTablet for tablet 7dcb2ce5ec9d46b48fd8a736e24897f0 (DEFAULT_TABLE table=test-table [id=07aa8cf91ec64877b634715622e88096]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:54:05.843034 4029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dcb2ce5ec9d46b48fd8a736e24897f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:54:05.843041 4169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dcb2ce5ec9d46b48fd8a736e24897f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:54:05.843086 4309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dcb2ce5ec9d46b48fd8a736e24897f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:54:05.844388 4399 tablet_bootstrap.cc:492] T 7dcb2ce5ec9d46b48fd8a736e24897f0 P af52e7ef760c43fb8df9f26dc88ed828: Bootstrap starting.
I20260928 21:54:05.844444 4421 ts_tablet_manager.cc:1918] T e86f14a768ca4a5a8f6d9dcbfa7efc12 P 59bb2db1feaa4da4b77ad2d68699feb1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:54:05.844504 4421 log.cc:1196] T e86f14a768ca4a5a8f6d9dcbfa7efc12 P 59bb2db1feaa4da4b77ad2d68699feb1: Deleting WAL directory at /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/minicluster-data/ts-0/wal/wals/e86f14a768ca4a5a8f6d9dcbfa7efc12
I20260928 21:54:05.844581 4398 tablet_bootstrap.cc:492] T 7dcb2ce5ec9d46b48fd8a736e24897f0 P deebb36055154ca88052129c5c7ed3bc: Bootstrap starting.
I20260928 21:54:05.844748 4421 ts_tablet_manager.cc:1939] T e86f14a768ca4a5a8f6d9dcbfa7efc12 P 59bb2db1feaa4da4b77ad2d68699feb1: Deleting consensus metadata
I20260928 21:54:05.844758 4399 tablet_bootstrap.cc:654] T 7dcb2ce5ec9d46b48fd8a736e24897f0 P af52e7ef760c43fb8df9f26dc88ed828: Neither blocks nor log segments found. Creating new log.
I20260928 21:54:05.844835 4422 ts_tablet_manager.cc:1905] T e86f14a768ca4a5a8f6d9dcbfa7efc12 P deebb36055154ca88052129c5c7ed3bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:54:05.844973 4398 tablet_bootstrap.cc:654] T 7dcb2ce5ec9d46b48fd8a736e24897f0 P deebb36055154ca88052129c5c7ed3bc: Neither blocks nor log segments found. Creating new log.
I20260928 21:54:05.845257 4400 tablet_bootstrap.cc:492] T 7dcb2ce5ec9d46b48fd8a736e24897f0 P 59bb2db1feaa4da4b77ad2d68699feb1: Bootstrap starting.
I20260928 21:54:05.845579 4399 tablet_bootstrap.cc:492] T 7dcb2ce5ec9d46b48fd8a736e24897f0 P af52e7ef760c43fb8df9f26dc88ed828: No bootstrap required, opened a new log
I20260928 21:54:05.845644 4400 tablet_bootstrap.cc:654] T 7dcb2ce5ec9d46b48fd8a736e24897f0 P 59bb2db1feaa4da4b77ad2d68699feb1: Neither blocks nor log segments found. Creating new log.
I20260928 21:54:05.845665 4399 ts_tablet_manager.cc:1397] T 7dcb2ce5ec9d46b48fd8a736e24897f0 P af52e7ef760c43fb8df9f26dc88ed828: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:54:05.845817 4399 raft_consensus.cc:348] T 7dcb2ce5ec9d46b48fd8a736e24897f0 P af52e7ef760c43fb8df9f26dc88ed828 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59bb2db1feaa4da4b77ad2d68699feb1" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 41757 } } peers { permanent_uuid: "deebb36055154ca88052129c5c7ed3bc" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 42025 } } peers { permanent_uuid: "af52e7ef760c43fb8df9f26dc88ed828" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 34801 } }
I20260928 21:54:05.845875 4399 raft_consensus.cc:374] T 7dcb2ce5ec9d46b48fd8a736e24897f0 P af52e7ef760c43fb8df9f26dc88ed828 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:54:05.845893 4399 raft_consensus.cc:729] T 7dcb2ce5ec9d46b48fd8a736e24897f0 P af52e7ef760c43fb8df9f26dc88ed828 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af52e7ef760c43fb8df9f26dc88ed828, State: Initialized, Role: FOLLOWER
I20260928 21:54:05.845932 4399 consensus_queue.cc:260] T 7dcb2ce5ec9d46b48fd8a736e24897f0 P af52e7ef760c43fb8df9f26dc88ed828 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59bb2db1feaa4da4b77ad2d68699feb1" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 41757 } } peers { permanent_uuid: "deebb36055154ca88052129c5c7ed3bc" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 42025 } } peers { permanent_uuid: "af52e7ef760c43fb8df9f26dc88ed828" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 34801 } }
I20260928 21:54:05.846011 4399 ts_tablet_manager.cc:1428] T 7dcb2ce5ec9d46b48fd8a736e24897f0 P af52e7ef760c43fb8df9f26dc88ed828: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:05.846058 3878 catalog_manager.cc:4841] TS 59bb2db1feaa4da4b77ad2d68699feb1 (127.2.182.65:41757): tablet e86f14a768ca4a5a8f6d9dcbfa7efc12 (table test-table [id=53f080f99be8477987bbeadd636fdbf4]) successfully deleted
I20260928 21:54:05.846588 4420 ts_tablet_manager.cc:1905] T e86f14a768ca4a5a8f6d9dcbfa7efc12 P af52e7ef760c43fb8df9f26dc88ed828: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:54:05.847692 4400 tablet_bootstrap.cc:492] T 7dcb2ce5ec9d46b48fd8a736e24897f0 P 59bb2db1feaa4da4b77ad2d68699feb1: No bootstrap required, opened a new log
I20260928 21:54:05.847747 4400 ts_tablet_manager.cc:1397] T 7dcb2ce5ec9d46b48fd8a736e24897f0 P 59bb2db1feaa4da4b77ad2d68699feb1: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 21:54:05.847891 4400 raft_consensus.cc:348] T 7dcb2ce5ec9d46b48fd8a736e24897f0 P 59bb2db1feaa4da4b77ad2d68699feb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59bb2db1feaa4da4b77ad2d68699feb1" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 41757 } } peers { permanent_uuid: "deebb36055154ca88052129c5c7ed3bc" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 42025 } } peers { permanent_uuid: "af52e7ef760c43fb8df9f26dc88ed828" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 34801 } }
I20260928 21:54:05.847970 4400 raft_consensus.cc:374] T 7dcb2ce5ec9d46b48fd8a736e24897f0 P 59bb2db1feaa4da4b77ad2d68699feb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:54:05.847990 4400 raft_consensus.cc:729] T 7dcb2ce5ec9d46b48fd8a736e24897f0 P 59bb2db1feaa4da4b77ad2d68699feb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59bb2db1feaa4da4b77ad2d68699feb1, State: Initialized, Role: FOLLOWER
I20260928 21:54:05.848042 4400 consensus_queue.cc:260] T 7dcb2ce5ec9d46b48fd8a736e24897f0 P 59bb2db1feaa4da4b77ad2d68699feb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59bb2db1feaa4da4b77ad2d68699feb1" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 41757 } } peers { permanent_uuid: "deebb36055154ca88052129c5c7ed3bc" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 42025 } } peers { permanent_uuid: "af52e7ef760c43fb8df9f26dc88ed828" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 34801 } }
I20260928 21:54:05.848126 4400 ts_tablet_manager.cc:1428] T 7dcb2ce5ec9d46b48fd8a736e24897f0 P 59bb2db1feaa4da4b77ad2d68699feb1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:05.848332 4398 tablet_bootstrap.cc:492] T 7dcb2ce5ec9d46b48fd8a736e24897f0 P deebb36055154ca88052129c5c7ed3bc: No bootstrap required, opened a new log
I20260928 21:54:05.848387 4398 ts_tablet_manager.cc:1397] T 7dcb2ce5ec9d46b48fd8a736e24897f0 P deebb36055154ca88052129c5c7ed3bc: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260928 21:54:05.848534 4398 raft_consensus.cc:348] T 7dcb2ce5ec9d46b48fd8a736e24897f0 P deebb36055154ca88052129c5c7ed3bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59bb2db1feaa4da4b77ad2d68699feb1" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 41757 } } peers { permanent_uuid: "deebb36055154ca88052129c5c7ed3bc" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 42025 } } peers { permanent_uuid: "af52e7ef760c43fb8df9f26dc88ed828" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 34801 } }
I20260928 21:54:05.848609 4398 raft_consensus.cc:374] T 7dcb2ce5ec9d46b48fd8a736e24897f0 P deebb36055154ca88052129c5c7ed3bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:54:05.848628 4398 raft_consensus.cc:729] T 7dcb2ce5ec9d46b48fd8a736e24897f0 P deebb36055154ca88052129c5c7ed3bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: deebb36055154ca88052129c5c7ed3bc, State: Initialized, Role: FOLLOWER
I20260928 21:54:05.848688 4398 consensus_queue.cc:260] T 7dcb2ce5ec9d46b48fd8a736e24897f0 P deebb36055154ca88052129c5c7ed3bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59bb2db1feaa4da4b77ad2d68699feb1" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 41757 } } peers { permanent_uuid: "deebb36055154ca88052129c5c7ed3bc" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 42025 } } peers { permanent_uuid: "af52e7ef760c43fb8df9f26dc88ed828" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 34801 } }
I20260928 21:54:05.848778 4398 ts_tablet_manager.cc:1428] T 7dcb2ce5ec9d46b48fd8a736e24897f0 P deebb36055154ca88052129c5c7ed3bc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:05.849735 4420 ts_tablet_manager.cc:1918] T e86f14a768ca4a5a8f6d9dcbfa7efc12 P af52e7ef760c43fb8df9f26dc88ed828: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:54:05.849812 4420 log.cc:1196] T e86f14a768ca4a5a8f6d9dcbfa7efc12 P af52e7ef760c43fb8df9f26dc88ed828: Deleting WAL directory at /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/minicluster-data/ts-2/wal/wals/e86f14a768ca4a5a8f6d9dcbfa7efc12
I20260928 21:54:05.850060 4420 ts_tablet_manager.cc:1939] T e86f14a768ca4a5a8f6d9dcbfa7efc12 P af52e7ef760c43fb8df9f26dc88ed828: Deleting consensus metadata
I20260928 21:54:05.850660 3879 catalog_manager.cc:4841] TS af52e7ef760c43fb8df9f26dc88ed828 (127.2.182.67:34801): tablet e86f14a768ca4a5a8f6d9dcbfa7efc12 (table test-table [id=53f080f99be8477987bbeadd636fdbf4]) successfully deleted
I20260928 21:54:05.851529 4422 ts_tablet_manager.cc:1918] T e86f14a768ca4a5a8f6d9dcbfa7efc12 P deebb36055154ca88052129c5c7ed3bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:54:05.851590 4422 log.cc:1196] T e86f14a768ca4a5a8f6d9dcbfa7efc12 P deebb36055154ca88052129c5c7ed3bc: Deleting WAL directory at /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/minicluster-data/ts-1/wal/wals/e86f14a768ca4a5a8f6d9dcbfa7efc12
I20260928 21:54:05.851799 4422 ts_tablet_manager.cc:1939] T e86f14a768ca4a5a8f6d9dcbfa7efc12 P deebb36055154ca88052129c5c7ed3bc: Deleting consensus metadata
I20260928 21:54:05.852151 3879 catalog_manager.cc:4841] TS deebb36055154ca88052129c5c7ed3bc (127.2.182.66:42025): tablet e86f14a768ca4a5a8f6d9dcbfa7efc12 (table test-table [id=53f080f99be8477987bbeadd636fdbf4]) successfully deleted
I20260928 21:54:05.857309 4408 raft_consensus.cc:482] T 7dcb2ce5ec9d46b48fd8a736e24897f0 P af52e7ef760c43fb8df9f26dc88ed828 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:54:05.857383 4408 raft_consensus.cc:504] T 7dcb2ce5ec9d46b48fd8a736e24897f0 P af52e7ef760c43fb8df9f26dc88ed828 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59bb2db1feaa4da4b77ad2d68699feb1" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 41757 } } peers { permanent_uuid: "deebb36055154ca88052129c5c7ed3bc" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 42025 } } peers { permanent_uuid: "af52e7ef760c43fb8df9f26dc88ed828" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 34801 } }
I20260928 21:54:05.857527 4408 leader_election.cc:288] T 7dcb2ce5ec9d46b48fd8a736e24897f0 P af52e7ef760c43fb8df9f26dc88ed828 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59bb2db1feaa4da4b77ad2d68699feb1 (127.2.182.65:41757), deebb36055154ca88052129c5c7ed3bc (127.2.182.66:42025)
I20260928 21:54:05.857693 4189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dcb2ce5ec9d46b48fd8a736e24897f0" candidate_uuid: "af52e7ef760c43fb8df9f26dc88ed828" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "deebb36055154ca88052129c5c7ed3bc" is_pre_election: true
I20260928 21:54:05.857715 4049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dcb2ce5ec9d46b48fd8a736e24897f0" candidate_uuid: "af52e7ef760c43fb8df9f26dc88ed828" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59bb2db1feaa4da4b77ad2d68699feb1" is_pre_election: true
I20260928 21:54:05.857781 4189 raft_consensus.cc:2454] T 7dcb2ce5ec9d46b48fd8a736e24897f0 P deebb36055154ca88052129c5c7ed3bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af52e7ef760c43fb8df9f26dc88ed828 in term 0.
I20260928 21:54:05.857797 4049 raft_consensus.cc:2454] T 7dcb2ce5ec9d46b48fd8a736e24897f0 P 59bb2db1feaa4da4b77ad2d68699feb1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af52e7ef760c43fb8df9f26dc88ed828 in term 0.
I20260928 21:54:05.857908 4263 leader_election.cc:302] T 7dcb2ce5ec9d46b48fd8a736e24897f0 P af52e7ef760c43fb8df9f26dc88ed828 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af52e7ef760c43fb8df9f26dc88ed828, deebb36055154ca88052129c5c7ed3bc; no voters:
I20260928 21:54:05.857998 4408 raft_consensus.cc:2781] T 7dcb2ce5ec9d46b48fd8a736e24897f0 P af52e7ef760c43fb8df9f26dc88ed828 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:54:05.858028 4408 raft_consensus.cc:482] T 7dcb2ce5ec9d46b48fd8a736e24897f0 P af52e7ef760c43fb8df9f26dc88ed828 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:54:05.858055 4408 raft_consensus.cc:3037] T 7dcb2ce5ec9d46b48fd8a736e24897f0 P af52e7ef760c43fb8df9f26dc88ed828 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:54:05.858716 4408 raft_consensus.cc:504] T 7dcb2ce5ec9d46b48fd8a736e24897f0 P af52e7ef760c43fb8df9f26dc88ed828 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59bb2db1feaa4da4b77ad2d68699feb1" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 41757 } } peers { permanent_uuid: "deebb36055154ca88052129c5c7ed3bc" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 42025 } } peers { permanent_uuid: "af52e7ef760c43fb8df9f26dc88ed828" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 34801 } }
I20260928 21:54:05.858829 4408 leader_election.cc:288] T 7dcb2ce5ec9d46b48fd8a736e24897f0 P af52e7ef760c43fb8df9f26dc88ed828 [CANDIDATE]: Term 1 election: Requested vote from peers 59bb2db1feaa4da4b77ad2d68699feb1 (127.2.182.65:41757), deebb36055154ca88052129c5c7ed3bc (127.2.182.66:42025)
I20260928 21:54:05.858970 4049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dcb2ce5ec9d46b48fd8a736e24897f0" candidate_uuid: "af52e7ef760c43fb8df9f26dc88ed828" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59bb2db1feaa4da4b77ad2d68699feb1"
I20260928 21:54:05.859040 4049 raft_consensus.cc:3037] T 7dcb2ce5ec9d46b48fd8a736e24897f0 P 59bb2db1feaa4da4b77ad2d68699feb1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:54:05.859035 4189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dcb2ce5ec9d46b48fd8a736e24897f0" candidate_uuid: "af52e7ef760c43fb8df9f26dc88ed828" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "deebb36055154ca88052129c5c7ed3bc"
I20260928 21:54:05.859102 4189 raft_consensus.cc:3037] T 7dcb2ce5ec9d46b48fd8a736e24897f0 P deebb36055154ca88052129c5c7ed3bc [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:54:05.859777 4189 raft_consensus.cc:2454] T 7dcb2ce5ec9d46b48fd8a736e24897f0 P deebb36055154ca88052129c5c7ed3bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af52e7ef760c43fb8df9f26dc88ed828 in term 1.
I20260928 21:54:05.859777 4049 raft_consensus.cc:2454] T 7dcb2ce5ec9d46b48fd8a736e24897f0 P 59bb2db1feaa4da4b77ad2d68699feb1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af52e7ef760c43fb8df9f26dc88ed828 in term 1.
I20260928 21:54:05.859967 4262 leader_election.cc:302] T 7dcb2ce5ec9d46b48fd8a736e24897f0 P af52e7ef760c43fb8df9f26dc88ed828 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59bb2db1feaa4da4b77ad2d68699feb1, af52e7ef760c43fb8df9f26dc88ed828; no voters:
I20260928 21:54:05.860054 4408 raft_consensus.cc:2781] T 7dcb2ce5ec9d46b48fd8a736e24897f0 P af52e7ef760c43fb8df9f26dc88ed828 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:54:05.860097 4408 raft_consensus.cc:686] T 7dcb2ce5ec9d46b48fd8a736e24897f0 P af52e7ef760c43fb8df9f26dc88ed828 [term 1 LEADER]: Becoming Leader. State: Replica: af52e7ef760c43fb8df9f26dc88ed828, State: Running, Role: LEADER
I20260928 21:54:05.860141 4408 consensus_queue.cc:237] T 7dcb2ce5ec9d46b48fd8a736e24897f0 P af52e7ef760c43fb8df9f26dc88ed828 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59bb2db1feaa4da4b77ad2d68699feb1" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 41757 } } peers { permanent_uuid: "deebb36055154ca88052129c5c7ed3bc" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 42025 } } peers { permanent_uuid: "af52e7ef760c43fb8df9f26dc88ed828" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 34801 } }
I20260928 21:54:05.860610 3891 catalog_manager.cc:5495] T 7dcb2ce5ec9d46b48fd8a736e24897f0 P af52e7ef760c43fb8df9f26dc88ed828 reported cstate change: term changed from 0 to 1, leader changed from <none> to af52e7ef760c43fb8df9f26dc88ed828 (127.2.182.67). New cstate: current_term: 1 leader_uuid: "af52e7ef760c43fb8df9f26dc88ed828" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59bb2db1feaa4da4b77ad2d68699feb1" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 41757 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "deebb36055154ca88052129c5c7ed3bc" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 42025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af52e7ef760c43fb8df9f26dc88ed828" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 34801 } health_report { overall_health: HEALTHY } } }
Sep 28 21:54:05 dist-test-slave-k6lz krb5kdc[3847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632445, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.126@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:54:05.900230 4432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:05.892074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:34929 (local address 127.0.0.1:54526)
0928 21:54:05.892218 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:05.892229 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:05.892296 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:05.892455 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:05.892456 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:05.893174 (+ 718us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:05.893228 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:05.893231 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:05.894276 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:05.894277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:05.894823 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:05.894825 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:05.895303 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:05.895304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:05.895391 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:05.895954 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:05.895958 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:05.897475 (+ 1517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:05.899072 (+ 1597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:05.899079 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:05.899081 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:05.899406 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:05.899598 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:05.899598 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:05.899599 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:05.899657 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:05.899936 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:05.899943 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:05.899992 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 21:54:05.900053 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 21:54:05.900149 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":75,"threads_started":1}
I20260928 21:54:05.900316 3941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:05.892149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54526 (local address 127.2.182.126:34929)
0928 21:54:05.892215 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:05.892217 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:05.892294 (+ 77us) server_negotiation.cc:407] Connection header received
0928 21:54:05.892349 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:05.892350 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:05.892370 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:05.892450 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:05.893331 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:05.894138 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:05.894911 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:05.895216 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:05.895283 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:05.897567 (+ 2284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:05.897570 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:05.897570 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:05.897578 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:05.898987 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:05.899474 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:05.899475 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:05.899475 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:05.899536 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:05.899713 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:05.899714 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:05.899715 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:05.899856 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:05.899925 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:05.900088 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 21:54:05.900215 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":32}
I20260928 21:54:05.902428 3891 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.2.182.126@KRBTEST.COM'} at 127.0.0.1:54526:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 21:54:05.902654 3891 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:54:05.904140 4029 tablet_service.cc:1467] Processing CreateTablet for tablet 2db1e7e66ba24d4aa7e15b8d2a17ae1d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1e3cf26f757042838285fdbd11fd7cc1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 21:54:05.904165 4169 tablet_service.cc:1467] Processing CreateTablet for tablet 2db1e7e66ba24d4aa7e15b8d2a17ae1d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1e3cf26f757042838285fdbd11fd7cc1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 21:54:05.904299 4029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2db1e7e66ba24d4aa7e15b8d2a17ae1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:54:05.904321 4169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2db1e7e66ba24d4aa7e15b8d2a17ae1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:54:05.905428 4400 tablet_bootstrap.cc:492] T 2db1e7e66ba24d4aa7e15b8d2a17ae1d P 59bb2db1feaa4da4b77ad2d68699feb1: Bootstrap starting.
I20260928 21:54:05.905512 4309 tablet_service.cc:1467] Processing CreateTablet for tablet 2db1e7e66ba24d4aa7e15b8d2a17ae1d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1e3cf26f757042838285fdbd11fd7cc1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 21:54:05.905632 4309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2db1e7e66ba24d4aa7e15b8d2a17ae1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:54:05.905836 4400 tablet_bootstrap.cc:654] T 2db1e7e66ba24d4aa7e15b8d2a17ae1d P 59bb2db1feaa4da4b77ad2d68699feb1: Neither blocks nor log segments found. Creating new log.
I20260928 21:54:05.905839 4398 tablet_bootstrap.cc:492] T 2db1e7e66ba24d4aa7e15b8d2a17ae1d P deebb36055154ca88052129c5c7ed3bc: Bootstrap starting.
I20260928 21:54:05.906234 4398 tablet_bootstrap.cc:654] T 2db1e7e66ba24d4aa7e15b8d2a17ae1d P deebb36055154ca88052129c5c7ed3bc: Neither blocks nor log segments found. Creating new log.
I20260928 21:54:05.907043 4399 tablet_bootstrap.cc:492] T 2db1e7e66ba24d4aa7e15b8d2a17ae1d P af52e7ef760c43fb8df9f26dc88ed828: Bootstrap starting.
I20260928 21:54:05.907240 4398 tablet_bootstrap.cc:492] T 2db1e7e66ba24d4aa7e15b8d2a17ae1d P deebb36055154ca88052129c5c7ed3bc: No bootstrap required, opened a new log
I20260928 21:54:05.907286 4398 ts_tablet_manager.cc:1397] T 2db1e7e66ba24d4aa7e15b8d2a17ae1d P deebb36055154ca88052129c5c7ed3bc: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:54:05.907373 4399 tablet_bootstrap.cc:654] T 2db1e7e66ba24d4aa7e15b8d2a17ae1d P af52e7ef760c43fb8df9f26dc88ed828: Neither blocks nor log segments found. Creating new log.
I20260928 21:54:05.907488 4400 tablet_bootstrap.cc:492] T 2db1e7e66ba24d4aa7e15b8d2a17ae1d P 59bb2db1feaa4da4b77ad2d68699feb1: No bootstrap required, opened a new log
I20260928 21:54:05.907529 4400 ts_tablet_manager.cc:1397] T 2db1e7e66ba24d4aa7e15b8d2a17ae1d P 59bb2db1feaa4da4b77ad2d68699feb1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:54:05.907483 4398 raft_consensus.cc:348] T 2db1e7e66ba24d4aa7e15b8d2a17ae1d P deebb36055154ca88052129c5c7ed3bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59bb2db1feaa4da4b77ad2d68699feb1" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 41757 } } peers { permanent_uuid: "deebb36055154ca88052129c5c7ed3bc" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 42025 } } peers { permanent_uuid: "af52e7ef760c43fb8df9f26dc88ed828" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 34801 } }
I20260928 21:54:05.907572 4398 raft_consensus.cc:374] T 2db1e7e66ba24d4aa7e15b8d2a17ae1d P deebb36055154ca88052129c5c7ed3bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:54:05.907595 4398 raft_consensus.cc:729] T 2db1e7e66ba24d4aa7e15b8d2a17ae1d P deebb36055154ca88052129c5c7ed3bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: deebb36055154ca88052129c5c7ed3bc, State: Initialized, Role: FOLLOWER
I20260928 21:54:05.907639 4398 consensus_queue.cc:260] T 2db1e7e66ba24d4aa7e15b8d2a17ae1d P deebb36055154ca88052129c5c7ed3bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59bb2db1feaa4da4b77ad2d68699feb1" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 41757 } } peers { permanent_uuid: "deebb36055154ca88052129c5c7ed3bc" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 42025 } } peers { permanent_uuid: "af52e7ef760c43fb8df9f26dc88ed828" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 34801 } }
I20260928 21:54:05.907665 4400 raft_consensus.cc:348] T 2db1e7e66ba24d4aa7e15b8d2a17ae1d P 59bb2db1feaa4da4b77ad2d68699feb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59bb2db1feaa4da4b77ad2d68699feb1" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 41757 } } peers { permanent_uuid: "deebb36055154ca88052129c5c7ed3bc" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 42025 } } peers { permanent_uuid: "af52e7ef760c43fb8df9f26dc88ed828" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 34801 } }
I20260928 21:54:05.907732 4400 raft_consensus.cc:374] T 2db1e7e66ba24d4aa7e15b8d2a17ae1d P 59bb2db1feaa4da4b77ad2d68699feb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:54:05.907729 4398 ts_tablet_manager.cc:1428] T 2db1e7e66ba24d4aa7e15b8d2a17ae1d P deebb36055154ca88052129c5c7ed3bc: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 21:54:05.907753 4400 raft_consensus.cc:729] T 2db1e7e66ba24d4aa7e15b8d2a17ae1d P 59bb2db1feaa4da4b77ad2d68699feb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59bb2db1feaa4da4b77ad2d68699feb1, State: Initialized, Role: FOLLOWER
I20260928 21:54:05.907806 4400 consensus_queue.cc:260] T 2db1e7e66ba24d4aa7e15b8d2a17ae1d P 59bb2db1feaa4da4b77ad2d68699feb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59bb2db1feaa4da4b77ad2d68699feb1" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 41757 } } peers { permanent_uuid: "deebb36055154ca88052129c5c7ed3bc" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 42025 } } peers { permanent_uuid: "af52e7ef760c43fb8df9f26dc88ed828" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 34801 } }
I20260928 21:54:05.907894 4400 ts_tablet_manager.cc:1428] T 2db1e7e66ba24d4aa7e15b8d2a17ae1d P 59bb2db1feaa4da4b77ad2d68699feb1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:05.907881 4404 tablet_replica.cc:439] T 2db1e7e66ba24d4aa7e15b8d2a17ae1d P deebb36055154ca88052129c5c7ed3bc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59bb2db1feaa4da4b77ad2d68699feb1" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 41757 } } peers { permanent_uuid: "deebb36055154ca88052129c5c7ed3bc" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 42025 } } peers { permanent_uuid: "af52e7ef760c43fb8df9f26dc88ed828" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 34801 } } }
I20260928 21:54:05.907959 4404 tablet_replica.cc:442] T 2db1e7e66ba24d4aa7e15b8d2a17ae1d P deebb36055154ca88052129c5c7ed3bc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:54:05.908020 4406 tablet_replica.cc:439] T 2db1e7e66ba24d4aa7e15b8d2a17ae1d P 59bb2db1feaa4da4b77ad2d68699feb1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59bb2db1feaa4da4b77ad2d68699feb1" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 41757 } } peers { permanent_uuid: "deebb36055154ca88052129c5c7ed3bc" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 42025 } } peers { permanent_uuid: "af52e7ef760c43fb8df9f26dc88ed828" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 34801 } } }
I20260928 21:54:05.908098 4406 tablet_replica.cc:442] T 2db1e7e66ba24d4aa7e15b8d2a17ae1d P 59bb2db1feaa4da4b77ad2d68699feb1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:54:05.908439 4399 tablet_bootstrap.cc:492] T 2db1e7e66ba24d4aa7e15b8d2a17ae1d P af52e7ef760c43fb8df9f26dc88ed828: No bootstrap required, opened a new log
I20260928 21:54:05.908485 4399 ts_tablet_manager.cc:1397] T 2db1e7e66ba24d4aa7e15b8d2a17ae1d P af52e7ef760c43fb8df9f26dc88ed828: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 21:54:05.908638 4399 raft_consensus.cc:348] T 2db1e7e66ba24d4aa7e15b8d2a17ae1d P af52e7ef760c43fb8df9f26dc88ed828 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59bb2db1feaa4da4b77ad2d68699feb1" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 41757 } } peers { permanent_uuid: "deebb36055154ca88052129c5c7ed3bc" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 42025 } } peers { permanent_uuid: "af52e7ef760c43fb8df9f26dc88ed828" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 34801 } }
I20260928 21:54:05.908699 4399 raft_consensus.cc:374] T 2db1e7e66ba24d4aa7e15b8d2a17ae1d P af52e7ef760c43fb8df9f26dc88ed828 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:54:05.908718 4399 raft_consensus.cc:729] T 2db1e7e66ba24d4aa7e15b8d2a17ae1d P af52e7ef760c43fb8df9f26dc88ed828 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af52e7ef760c43fb8df9f26dc88ed828, State: Initialized, Role: FOLLOWER
I20260928 21:54:05.908759 4399 consensus_queue.cc:260] T 2db1e7e66ba24d4aa7e15b8d2a17ae1d P af52e7ef760c43fb8df9f26dc88ed828 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59bb2db1feaa4da4b77ad2d68699feb1" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 41757 } } peers { permanent_uuid: "deebb36055154ca88052129c5c7ed3bc" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 42025 } } peers { permanent_uuid: "af52e7ef760c43fb8df9f26dc88ed828" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 34801 } }
I20260928 21:54:05.908840 4399 ts_tablet_manager.cc:1428] T 2db1e7e66ba24d4aa7e15b8d2a17ae1d P af52e7ef760c43fb8df9f26dc88ed828: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:05.908860 4408 tablet_replica.cc:439] T 2db1e7e66ba24d4aa7e15b8d2a17ae1d P af52e7ef760c43fb8df9f26dc88ed828: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59bb2db1feaa4da4b77ad2d68699feb1" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 41757 } } peers { permanent_uuid: "deebb36055154ca88052129c5c7ed3bc" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 42025 } } peers { permanent_uuid: "af52e7ef760c43fb8df9f26dc88ed828" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 34801 } } }
I20260928 21:54:05.908951 4408 tablet_replica.cc:442] T 2db1e7e66ba24d4aa7e15b8d2a17ae1d P af52e7ef760c43fb8df9f26dc88ed828: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:54:05.909924 4408 raft_consensus.cc:482] T 2db1e7e66ba24d4aa7e15b8d2a17ae1d P af52e7ef760c43fb8df9f26dc88ed828 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:54:05.909978 4408 raft_consensus.cc:504] T 2db1e7e66ba24d4aa7e15b8d2a17ae1d P af52e7ef760c43fb8df9f26dc88ed828 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59bb2db1feaa4da4b77ad2d68699feb1" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 41757 } } peers { permanent_uuid: "deebb36055154ca88052129c5c7ed3bc" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 42025 } } peers { permanent_uuid: "af52e7ef760c43fb8df9f26dc88ed828" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 34801 } }
I20260928 21:54:05.910092 4408 leader_election.cc:288] T 2db1e7e66ba24d4aa7e15b8d2a17ae1d P af52e7ef760c43fb8df9f26dc88ed828 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59bb2db1feaa4da4b77ad2d68699feb1 (127.2.182.65:41757), deebb36055154ca88052129c5c7ed3bc (127.2.182.66:42025)
I20260928 21:54:05.910270 4049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db1e7e66ba24d4aa7e15b8d2a17ae1d" candidate_uuid: "af52e7ef760c43fb8df9f26dc88ed828" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59bb2db1feaa4da4b77ad2d68699feb1" is_pre_election: true
I20260928 21:54:05.910267 4189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db1e7e66ba24d4aa7e15b8d2a17ae1d" candidate_uuid: "af52e7ef760c43fb8df9f26dc88ed828" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "deebb36055154ca88052129c5c7ed3bc" is_pre_election: true
I20260928 21:54:05.910359 4189 raft_consensus.cc:2454] T 2db1e7e66ba24d4aa7e15b8d2a17ae1d P deebb36055154ca88052129c5c7ed3bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af52e7ef760c43fb8df9f26dc88ed828 in term 0.
I20260928 21:54:05.910360 4049 raft_consensus.cc:2454] T 2db1e7e66ba24d4aa7e15b8d2a17ae1d P 59bb2db1feaa4da4b77ad2d68699feb1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af52e7ef760c43fb8df9f26dc88ed828 in term 0.
I20260928 21:54:05.910487 4262 leader_election.cc:302] T 2db1e7e66ba24d4aa7e15b8d2a17ae1d P af52e7ef760c43fb8df9f26dc88ed828 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59bb2db1feaa4da4b77ad2d68699feb1, af52e7ef760c43fb8df9f26dc88ed828; no voters:
I20260928 21:54:05.910557 4408 raft_consensus.cc:2781] T 2db1e7e66ba24d4aa7e15b8d2a17ae1d P af52e7ef760c43fb8df9f26dc88ed828 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:54:05.910586 4408 raft_consensus.cc:482] T 2db1e7e66ba24d4aa7e15b8d2a17ae1d P af52e7ef760c43fb8df9f26dc88ed828 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:54:05.910612 4408 raft_consensus.cc:3037] T 2db1e7e66ba24d4aa7e15b8d2a17ae1d P af52e7ef760c43fb8df9f26dc88ed828 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:54:05.911168 4408 raft_consensus.cc:504] T 2db1e7e66ba24d4aa7e15b8d2a17ae1d P af52e7ef760c43fb8df9f26dc88ed828 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59bb2db1feaa4da4b77ad2d68699feb1" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 41757 } } peers { permanent_uuid: "deebb36055154ca88052129c5c7ed3bc" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 42025 } } peers { permanent_uuid: "af52e7ef760c43fb8df9f26dc88ed828" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 34801 } }
I20260928 21:54:05.911275 4408 leader_election.cc:288] T 2db1e7e66ba24d4aa7e15b8d2a17ae1d P af52e7ef760c43fb8df9f26dc88ed828 [CANDIDATE]: Term 1 election: Requested vote from peers 59bb2db1feaa4da4b77ad2d68699feb1 (127.2.182.65:41757), deebb36055154ca88052129c5c7ed3bc (127.2.182.66:42025)
I20260928 21:54:05.911485 4189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db1e7e66ba24d4aa7e15b8d2a17ae1d" candidate_uuid: "af52e7ef760c43fb8df9f26dc88ed828" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "deebb36055154ca88052129c5c7ed3bc"
I20260928 21:54:05.911499 4049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db1e7e66ba24d4aa7e15b8d2a17ae1d" candidate_uuid: "af52e7ef760c43fb8df9f26dc88ed828" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59bb2db1feaa4da4b77ad2d68699feb1"
I20260928 21:54:05.911561 4049 raft_consensus.cc:3037] T 2db1e7e66ba24d4aa7e15b8d2a17ae1d P 59bb2db1feaa4da4b77ad2d68699feb1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:54:05.911556 4189 raft_consensus.cc:3037] T 2db1e7e66ba24d4aa7e15b8d2a17ae1d P deebb36055154ca88052129c5c7ed3bc [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:54:05.912074 4049 raft_consensus.cc:2454] T 2db1e7e66ba24d4aa7e15b8d2a17ae1d P 59bb2db1feaa4da4b77ad2d68699feb1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af52e7ef760c43fb8df9f26dc88ed828 in term 1.
I20260928 21:54:05.912217 4262 leader_election.cc:302] T 2db1e7e66ba24d4aa7e15b8d2a17ae1d P af52e7ef760c43fb8df9f26dc88ed828 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59bb2db1feaa4da4b77ad2d68699feb1, af52e7ef760c43fb8df9f26dc88ed828; no voters:
I20260928 21:54:05.912259 4189 raft_consensus.cc:2454] T 2db1e7e66ba24d4aa7e15b8d2a17ae1d P deebb36055154ca88052129c5c7ed3bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af52e7ef760c43fb8df9f26dc88ed828 in term 1.
I20260928 21:54:05.912302 4408 raft_consensus.cc:2781] T 2db1e7e66ba24d4aa7e15b8d2a17ae1d P af52e7ef760c43fb8df9f26dc88ed828 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:54:05.912340 4408 raft_consensus.cc:686] T 2db1e7e66ba24d4aa7e15b8d2a17ae1d P af52e7ef760c43fb8df9f26dc88ed828 [term 1 LEADER]: Becoming Leader. State: Replica: af52e7ef760c43fb8df9f26dc88ed828, State: Running, Role: LEADER
I20260928 21:54:05.912375 4408 consensus_queue.cc:237] T 2db1e7e66ba24d4aa7e15b8d2a17ae1d P af52e7ef760c43fb8df9f26dc88ed828 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59bb2db1feaa4da4b77ad2d68699feb1" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 41757 } } peers { permanent_uuid: "deebb36055154ca88052129c5c7ed3bc" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 42025 } } peers { permanent_uuid: "af52e7ef760c43fb8df9f26dc88ed828" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 34801 } }
I20260928 21:54:05.912667 4408 tablet_replica.cc:439] T 2db1e7e66ba24d4aa7e15b8d2a17ae1d P af52e7ef760c43fb8df9f26dc88ed828: TxnStatusTablet state changed. Reason: New leader af52e7ef760c43fb8df9f26dc88ed828. Latest consensus state: current_term: 1 leader_uuid: "af52e7ef760c43fb8df9f26dc88ed828" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59bb2db1feaa4da4b77ad2d68699feb1" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 41757 } } peers { permanent_uuid: "deebb36055154ca88052129c5c7ed3bc" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 42025 } } peers { permanent_uuid: "af52e7ef760c43fb8df9f26dc88ed828" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 34801 } } }
I20260928 21:54:05.912748 4408 tablet_replica.cc:442] T 2db1e7e66ba24d4aa7e15b8d2a17ae1d P af52e7ef760c43fb8df9f26dc88ed828: This TxnStatusTablet replica's current role is: LEADER
W20260928 21:54:05.912936 4376 tablet.cc:2369] T 2db1e7e66ba24d4aa7e15b8d2a17ae1d P af52e7ef760c43fb8df9f26dc88ed828: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:54:05.912984 3892 catalog_manager.cc:5495] T 2db1e7e66ba24d4aa7e15b8d2a17ae1d P af52e7ef760c43fb8df9f26dc88ed828 reported cstate change: term changed from 0 to 1, leader changed from <none> to af52e7ef760c43fb8df9f26dc88ed828 (127.2.182.67). New cstate: current_term: 1 leader_uuid: "af52e7ef760c43fb8df9f26dc88ed828" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59bb2db1feaa4da4b77ad2d68699feb1" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 41757 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "deebb36055154ca88052129c5c7ed3bc" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 42025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af52e7ef760c43fb8df9f26dc88ed828" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 34801 } health_report { overall_health: HEALTHY } } }
I20260928 21:54:05.913273 4437 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 21:54:05.913380 4437 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 21:54:05.918782 4432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:05.916074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.67:34801 (local address 127.0.0.1:56736)
0928 21:54:05.916135 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:05.916146 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:05.916188 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:05.916339 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:05.916339 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:05.916340 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 21:54:05.916394 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:05.916396 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:05.917243 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:05.917243 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:05.917822 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:05.917824 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:05.918272 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:05.918273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:05.918337 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:05.918352 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 21:54:05.918658 (+ 306us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 21:54:05.918661 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:54:05.918690 (+ 29us) client_negotiation.cc:240] Negotiation successful
0928 21:54:05.918726 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 21:54:05.918792 4394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:05.916140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56736 (local address 127.2.182.67:34801)
0928 21:54:05.916181 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:05.916183 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:05.916193 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:54:05.916248 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:05.916249 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:05.916259 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:05.916312 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:54:05.916449 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:05.917158 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:05.917905 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:05.918192 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:05.918276 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:05.918563 (+ 287us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:54:05.918600 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:54:05.918656 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:05.918696 (+ 40us) server_negotiation.cc:299] Negotiation successful
0928 21:54:05.918719 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 21:54:05.919852 4189 raft_consensus.cc:1261] T 2db1e7e66ba24d4aa7e15b8d2a17ae1d P deebb36055154ca88052129c5c7ed3bc [term 1 FOLLOWER]: Refusing update from remote peer af52e7ef760c43fb8df9f26dc88ed828: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:54:05.919929 4049 raft_consensus.cc:1261] T 2db1e7e66ba24d4aa7e15b8d2a17ae1d P 59bb2db1feaa4da4b77ad2d68699feb1 [term 1 FOLLOWER]: Refusing update from remote peer af52e7ef760c43fb8df9f26dc88ed828: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:54:05.920150 4405 consensus_queue.cc:1035] T 2db1e7e66ba24d4aa7e15b8d2a17ae1d P af52e7ef760c43fb8df9f26dc88ed828 [LEADER]: Connected to new peer: Peer: permanent_uuid: "deebb36055154ca88052129c5c7ed3bc" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 42025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:54:05.920275 4408 consensus_queue.cc:1035] T 2db1e7e66ba24d4aa7e15b8d2a17ae1d P af52e7ef760c43fb8df9f26dc88ed828 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59bb2db1feaa4da4b77ad2d68699feb1" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 41757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:54:05.920717 4404 tablet_replica.cc:439] T 2db1e7e66ba24d4aa7e15b8d2a17ae1d P deebb36055154ca88052129c5c7ed3bc: TxnStatusTablet state changed. Reason: New leader af52e7ef760c43fb8df9f26dc88ed828. Latest consensus state: current_term: 1 leader_uuid: "af52e7ef760c43fb8df9f26dc88ed828" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59bb2db1feaa4da4b77ad2d68699feb1" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 41757 } } peers { permanent_uuid: "deebb36055154ca88052129c5c7ed3bc" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 42025 } } peers { permanent_uuid: "af52e7ef760c43fb8df9f26dc88ed828" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 34801 } } }
I20260928 21:54:05.920828 4404 tablet_replica.cc:442] T 2db1e7e66ba24d4aa7e15b8d2a17ae1d P deebb36055154ca88052129c5c7ed3bc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:54:05.920864 4406 tablet_replica.cc:439] T 2db1e7e66ba24d4aa7e15b8d2a17ae1d P 59bb2db1feaa4da4b77ad2d68699feb1: TxnStatusTablet state changed. Reason: New leader af52e7ef760c43fb8df9f26dc88ed828. Latest consensus state: current_term: 1 leader_uuid: "af52e7ef760c43fb8df9f26dc88ed828" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59bb2db1feaa4da4b77ad2d68699feb1" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 41757 } } peers { permanent_uuid: "deebb36055154ca88052129c5c7ed3bc" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 42025 } } peers { permanent_uuid: "af52e7ef760c43fb8df9f26dc88ed828" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 34801 } } }
I20260928 21:54:05.920950 4406 tablet_replica.cc:442] T 2db1e7e66ba24d4aa7e15b8d2a17ae1d P 59bb2db1feaa4da4b77ad2d68699feb1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:54:05.921341 4408 tablet_replica.cc:439] T 2db1e7e66ba24d4aa7e15b8d2a17ae1d P af52e7ef760c43fb8df9f26dc88ed828: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "af52e7ef760c43fb8df9f26dc88ed828" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59bb2db1feaa4da4b77ad2d68699feb1" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 41757 } } peers { permanent_uuid: "deebb36055154ca88052129c5c7ed3bc" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 42025 } } peers { permanent_uuid: "af52e7ef760c43fb8df9f26dc88ed828" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 34801 } } }
I20260928 21:54:05.921411 4408 tablet_replica.cc:442] T 2db1e7e66ba24d4aa7e15b8d2a17ae1d P af52e7ef760c43fb8df9f26dc88ed828: This TxnStatusTablet replica's current role is: LEADER
I20260928 21:54:05.921670 4408 tablet_replica.cc:439] T 2db1e7e66ba24d4aa7e15b8d2a17ae1d P af52e7ef760c43fb8df9f26dc88ed828: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "af52e7ef760c43fb8df9f26dc88ed828" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59bb2db1feaa4da4b77ad2d68699feb1" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 41757 } } peers { permanent_uuid: "deebb36055154ca88052129c5c7ed3bc" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 42025 } } peers { permanent_uuid: "af52e7ef760c43fb8df9f26dc88ed828" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 34801 } } }
I20260928 21:54:05.921741 4408 tablet_replica.cc:442] T 2db1e7e66ba24d4aa7e15b8d2a17ae1d P af52e7ef760c43fb8df9f26dc88ed828: This TxnStatusTablet replica's current role is: LEADER
I20260928 21:54:05.922012 4412 mvcc.cc:205] Tried to move back new op lower bound from 7334430498486247424 to 7334430498457436160. Current Snapshot: MvccSnapshot[applied={T|T < 7334430498486247424}]
I20260928 21:54:05.922106 4406 tablet_replica.cc:439] T 2db1e7e66ba24d4aa7e15b8d2a17ae1d P 59bb2db1feaa4da4b77ad2d68699feb1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "af52e7ef760c43fb8df9f26dc88ed828" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59bb2db1feaa4da4b77ad2d68699feb1" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 41757 } } peers { permanent_uuid: "deebb36055154ca88052129c5c7ed3bc" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 42025 } } peers { permanent_uuid: "af52e7ef760c43fb8df9f26dc88ed828" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 34801 } } }
I20260928 21:54:05.922174 4406 tablet_replica.cc:442] T 2db1e7e66ba24d4aa7e15b8d2a17ae1d P 59bb2db1feaa4da4b77ad2d68699feb1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:54:05.922271 4404 tablet_replica.cc:439] T 2db1e7e66ba24d4aa7e15b8d2a17ae1d P deebb36055154ca88052129c5c7ed3bc: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "af52e7ef760c43fb8df9f26dc88ed828" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59bb2db1feaa4da4b77ad2d68699feb1" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 41757 } } peers { permanent_uuid: "deebb36055154ca88052129c5c7ed3bc" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 42025 } } peers { permanent_uuid: "af52e7ef760c43fb8df9f26dc88ed828" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 34801 } } }
I20260928 21:54:05.922364 4404 tablet_replica.cc:442] T 2db1e7e66ba24d4aa7e15b8d2a17ae1d P deebb36055154ca88052129c5c7ed3bc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:54:05.927770 4381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:05.925119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.67:34801 (local address 127.2.182.67:43931)
0928 21:54:05.925174 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:05.925184 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:05.925222 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:05.925432 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:05.925432 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:05.925433 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 21:54:05.925491 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:05.925493 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:05.926333 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:05.926334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:05.926909 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:05.926910 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:05.927398 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:05.927399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:05.927460 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:05.927473 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 21:54:05.927632 (+ 159us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 21:54:05.927633 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 21:54:05.927661 (+ 28us) client_negotiation.cc:240] Negotiation successful
0928 21:54:05.927697 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260928 21:54:05.927778 4394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:05.925275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.67:43931 (local address 127.2.182.67:34801)
0928 21:54:05.925330 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:05.925333 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:05.925341 (+ 8us) server_negotiation.cc:407] Connection header received
0928 21:54:05.925361 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:05.925362 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:05.925371 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:05.925428 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:54:05.925534 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:05.926262 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:05.926991 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:05.927278 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:05.927391 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:05.927546 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:54:05.927578 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:54:05.927629 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:05.927667 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 21:54:05.927685 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":36}
I20260928 21:54:05.929891 4049 raft_consensus.cc:1261] T 7dcb2ce5ec9d46b48fd8a736e24897f0 P 59bb2db1feaa4da4b77ad2d68699feb1 [term 1 FOLLOWER]: Refusing update from remote peer af52e7ef760c43fb8df9f26dc88ed828: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:54:05.929927 4189 raft_consensus.cc:1261] T 7dcb2ce5ec9d46b48fd8a736e24897f0 P deebb36055154ca88052129c5c7ed3bc [term 1 FOLLOWER]: Refusing update from remote peer af52e7ef760c43fb8df9f26dc88ed828: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:54:05.930225 4408 consensus_queue.cc:1035] T 7dcb2ce5ec9d46b48fd8a736e24897f0 P af52e7ef760c43fb8df9f26dc88ed828 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59bb2db1feaa4da4b77ad2d68699feb1" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 41757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:54:05.930346 4408 consensus_queue.cc:1035] T 7dcb2ce5ec9d46b48fd8a736e24897f0 P af52e7ef760c43fb8df9f26dc88ed828 [LEADER]: Connected to new peer: Peer: permanent_uuid: "deebb36055154ca88052129c5c7ed3bc" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 42025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:54:05.940644 3892 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54520:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:54:05.940744 3892 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54520:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:54:05.941308 3892 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 43159bea0e58402096e6d48ceadc44c1: Sending DeleteTablet for 3 replicas of tablet 7dcb2ce5ec9d46b48fd8a736e24897f0
I20260928 21:54:05.941514 4029 tablet_service.cc:1514] Processing DeleteTablet for tablet 7dcb2ce5ec9d46b48fd8a736e24897f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:54:05 UTC) from {username='kudu', principal='kudu/127.2.182.126@KRBTEST.COM'} at 127.0.0.1:56938
I20260928 21:54:05.941524 4169 tablet_service.cc:1514] Processing DeleteTablet for tablet 7dcb2ce5ec9d46b48fd8a736e24897f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:54:05 UTC) from {username='kudu', principal='kudu/127.2.182.126@KRBTEST.COM'} at 127.0.0.1:48384
I20260928 21:54:05.941604 4421 tablet_replica.cc:330] T 7dcb2ce5ec9d46b48fd8a736e24897f0 P 59bb2db1feaa4da4b77ad2d68699feb1: stopping tablet replica
I20260928 21:54:05.941623 4422 tablet_replica.cc:330] T 7dcb2ce5ec9d46b48fd8a736e24897f0 P deebb36055154ca88052129c5c7ed3bc: stopping tablet replica
I20260928 21:54:05.941642 4421 raft_consensus.cc:2229] T 7dcb2ce5ec9d46b48fd8a736e24897f0 P 59bb2db1feaa4da4b77ad2d68699feb1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:54:05.941661 4422 raft_consensus.cc:2229] T 7dcb2ce5ec9d46b48fd8a736e24897f0 P deebb36055154ca88052129c5c7ed3bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:54:05.941673 4421 raft_consensus.cc:2258] T 7dcb2ce5ec9d46b48fd8a736e24897f0 P 59bb2db1feaa4da4b77ad2d68699feb1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:54:05.941692 4422 raft_consensus.cc:2258] T 7dcb2ce5ec9d46b48fd8a736e24897f0 P deebb36055154ca88052129c5c7ed3bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:54:05.941923 4309 tablet_service.cc:1514] Processing DeleteTablet for tablet 7dcb2ce5ec9d46b48fd8a736e24897f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:54:05 UTC) from {username='kudu', principal='kudu/127.2.182.126@KRBTEST.COM'} at 127.0.0.1:56710
I20260928 21:54:05.942018 4420 tablet_replica.cc:330] T 7dcb2ce5ec9d46b48fd8a736e24897f0 P af52e7ef760c43fb8df9f26dc88ed828: stopping tablet replica
I20260928 21:54:05.942070 4420 raft_consensus.cc:2229] T 7dcb2ce5ec9d46b48fd8a736e24897f0 P af52e7ef760c43fb8df9f26dc88ed828 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:54:05.942118 4420 raft_consensus.cc:2258] T 7dcb2ce5ec9d46b48fd8a736e24897f0 P af52e7ef760c43fb8df9f26dc88ed828 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:54:05.942185 4422 ts_tablet_manager.cc:1905] T 7dcb2ce5ec9d46b48fd8a736e24897f0 P deebb36055154ca88052129c5c7ed3bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:54:05.942265 4421 ts_tablet_manager.cc:1905] T 7dcb2ce5ec9d46b48fd8a736e24897f0 P 59bb2db1feaa4da4b77ad2d68699feb1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:54:05.943920 4420 ts_tablet_manager.cc:1905] T 7dcb2ce5ec9d46b48fd8a736e24897f0 P af52e7ef760c43fb8df9f26dc88ed828: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:54:05.944188 4421 ts_tablet_manager.cc:1918] T 7dcb2ce5ec9d46b48fd8a736e24897f0 P 59bb2db1feaa4da4b77ad2d68699feb1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:54:05.944180 4422 ts_tablet_manager.cc:1918] T 7dcb2ce5ec9d46b48fd8a736e24897f0 P deebb36055154ca88052129c5c7ed3bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:54:05.944231 4422 log.cc:1196] T 7dcb2ce5ec9d46b48fd8a736e24897f0 P deebb36055154ca88052129c5c7ed3bc: Deleting WAL directory at /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/minicluster-data/ts-1/wal/wals/7dcb2ce5ec9d46b48fd8a736e24897f0
I20260928 21:54:05.944231 4421 log.cc:1196] T 7dcb2ce5ec9d46b48fd8a736e24897f0 P 59bb2db1feaa4da4b77ad2d68699feb1: Deleting WAL directory at /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/minicluster-data/ts-0/wal/wals/7dcb2ce5ec9d46b48fd8a736e24897f0
I20260928 21:54:05.944438 4422 ts_tablet_manager.cc:1939] T 7dcb2ce5ec9d46b48fd8a736e24897f0 P deebb36055154ca88052129c5c7ed3bc: Deleting consensus metadata
I20260928 21:54:05.944438 4421 ts_tablet_manager.cc:1939] T 7dcb2ce5ec9d46b48fd8a736e24897f0 P 59bb2db1feaa4da4b77ad2d68699feb1: Deleting consensus metadata
I20260928 21:54:05.944722 3878 catalog_manager.cc:4841] TS 59bb2db1feaa4da4b77ad2d68699feb1 (127.2.182.65:41757): tablet 7dcb2ce5ec9d46b48fd8a736e24897f0 (table test-table [id=07aa8cf91ec64877b634715622e88096]) successfully deleted
I20260928 21:54:05.944816 3879 catalog_manager.cc:4841] TS deebb36055154ca88052129c5c7ed3bc (127.2.182.66:42025): tablet 7dcb2ce5ec9d46b48fd8a736e24897f0 (table test-table [id=07aa8cf91ec64877b634715622e88096]) successfully deleted
I20260928 21:54:05.945477 4420 ts_tablet_manager.cc:1918] T 7dcb2ce5ec9d46b48fd8a736e24897f0 P af52e7ef760c43fb8df9f26dc88ed828: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:54:05.945523 4420 log.cc:1196] T 7dcb2ce5ec9d46b48fd8a736e24897f0 P af52e7ef760c43fb8df9f26dc88ed828: Deleting WAL directory at /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632440055392-2777-0/minicluster-data/ts-2/wal/wals/7dcb2ce5ec9d46b48fd8a736e24897f0
I20260928 21:54:05.945703 4420 ts_tablet_manager.cc:1939] T 7dcb2ce5ec9d46b48fd8a736e24897f0 P af52e7ef760c43fb8df9f26dc88ed828: Deleting consensus metadata
I20260928 21:54:05.946098 3879 catalog_manager.cc:4841] TS af52e7ef760c43fb8df9f26dc88ed828 (127.2.182.67:34801): tablet 7dcb2ce5ec9d46b48fd8a736e24897f0 (table test-table [id=07aa8cf91ec64877b634715622e88096]) successfully deleted
Sep 28 21:54:05 dist-test-slave-k6lz krb5kdc[3847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632445, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.182.65@KRBTEST.COM
I20260928 21:54:05.953019 4395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:05.943909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56950 (local address 127.2.182.65:41757)
0928 21:54:05.943955 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:05.943958 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:05.944072 (+ 114us) server_negotiation.cc:407] Connection header received
0928 21:54:05.944742 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:05.944743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:05.944754 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:05.944782 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:05.945397 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:05.945884 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:05.946662 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:05.946873 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:05.946928 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:05.949803 (+ 2875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:05.949815 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:05.949815 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:05.949824 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:05.951727 (+ 1903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:05.952052 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:05.952056 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:05.952056 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:05.952123 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:05.952363 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:05.952363 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:05.952364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:05.952558 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:05.952739 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:05.952862 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 21:54:05.952950 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 21:54:05.953527 4089 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56950
I20260928 21:54:05.960150 3941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:05.954614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54538 (local address 127.2.182.126:34929)
0928 21:54:05.954669 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:05.954671 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:05.954681 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:54:05.954727 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:05.954728 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:05.954738 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:05.954788 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:05.955262 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:05.955813 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:05.956465 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:05.956666 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:05.956721 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:05.957839 (+ 1118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:05.957841 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:05.957841 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:05.957848 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:05.959116 (+ 1268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:05.959469 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:05.959470 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:05.959471 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:05.959522 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:05.959679 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:05.959680 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:05.959680 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:05.959798 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:05.959884 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:05.959981 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 21:54:05.960057 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
W20260928 21:54:05.960372 3892 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54538
I20260928 21:54:05.960944 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 3964
I20260928 21:54:05.966032 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 4104
I20260928 21:54:05.970825 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 4244
I20260928 21:54:05.976300 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 3860
2026-09-28T21:54:05Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2041 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:54:06 dist-test-slave-k6lz krb5kdc[4453](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:54:06 dist-test-slave-k6lz krb5kdc[4453](info): set up 2 sockets
Sep 28 21:54:06 dist-test-slave-k6lz krb5kdc[4453](info): commencing operation
krb5kdc: starting...
W20260928 21:54:06.938939 2777 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Sep 28 21:54:07 dist-test-slave-k6lz krb5kdc[4453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632447, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:54:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:54:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab.
Entry for principal kudu/127.2.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab.
I20260928 21:54:07.054384 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:44601
--webserver_interface=127.2.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab
--principal=kudu/127.2.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:33301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.126:44601
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:54:07.135469 4466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:54:07.135697 4466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:54:07.135738 4466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:54:07.136960 4466 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:54:07.137002 4466 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 21:54:07.137018 4466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:54:07.137035 4466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:54:07.137050 4466 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:54:07.137068 4466 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:54:07.138624 4466 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:33301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.126:44601
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab
--principal=kudu/127.2.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:44601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:54:07.138904 4466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:54:07.139118 4466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:54:07.144132 4472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:07.144152 4474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:07.144140 4471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:07.144366 4466 server_base.cc:1034] running on GCE node
I20260928 21:54:07.144629 4466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:07.144942 4466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:07.146106 4466 hybrid_clock.cc:648] HybridClock initialized: now 1790632447146087 us; error 35 us; skew 500 ppm
Sep 28 21:54:07 dist-test-slave-k6lz krb5kdc[4453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632447, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:54:07.153927 4466 init.cc:377] Logged in from keytab as kudu/127.2.182.126@KRBTEST.COM (short username kudu)
I20260928 21:54:07.154474 4466 webserver.cc:460] Webserver started at http://127.2.182.126:32873/ using document root <none> and password file <none>
I20260928 21:54:07.154742 4466 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:07.154793 4466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:07.154980 4466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:54:07.156179 4466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/minicluster-data/master-0/data/instance:
uuid: "39ad015d374848d084369d0f182d3f51"
format_stamp: "Formatted at 2026-09-28 21:54:07 on dist-test-slave-k6lz"
server_key: "17d39e617b71925f14fb3970993d5b7c"
server_key_iv: "8df45154b71f2069fe0b723ccd1fb055"
server_key_version: "encryptionkey@0"
I20260928 21:54:07.156529 4466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/minicluster-data/master-0/wal/instance:
uuid: "39ad015d374848d084369d0f182d3f51"
format_stamp: "Formatted at 2026-09-28 21:54:07 on dist-test-slave-k6lz"
server_key: "17d39e617b71925f14fb3970993d5b7c"
server_key_iv: "8df45154b71f2069fe0b723ccd1fb055"
server_key_version: "encryptionkey@0"
I20260928 21:54:07.157908 4466 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:54:07.158769 4481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:07.159005 4466 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:54:07.159106 4466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/minicluster-data/master-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/minicluster-data/master-0/wal
uuid: "39ad015d374848d084369d0f182d3f51"
format_stamp: "Formatted at 2026-09-28 21:54:07 on dist-test-slave-k6lz"
server_key: "17d39e617b71925f14fb3970993d5b7c"
server_key_iv: "8df45154b71f2069fe0b723ccd1fb055"
server_key_version: "encryptionkey@0"
I20260928 21:54:07.159173 4466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:07.184800 4466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:54:07.186709 4466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:54:07.186844 4466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:54:07.191741 4466 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.126:44601
I20260928 21:54:07.191783 4543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.126:44601 every 8 connection(s)
I20260928 21:54:07.192111 4466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
I20260928 21:54:07.192719 4544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:54:07.194955 4544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39ad015d374848d084369d0f182d3f51: Bootstrap starting.
I20260928 21:54:07.195590 4544 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 39ad015d374848d084369d0f182d3f51: Neither blocks nor log segments found. Creating new log.
I20260928 21:54:07.195837 4544 log.cc:824] T 00000000000000000000000000000000 P 39ad015d374848d084369d0f182d3f51: Log is configured to *not* fsync() on all Append() calls
I20260928 21:54:07.196748 4544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39ad015d374848d084369d0f182d3f51: No bootstrap required, opened a new log
I20260928 21:54:07.198920 4544 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39ad015d374848d084369d0f182d3f51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39ad015d374848d084369d0f182d3f51" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 44601 } }
I20260928 21:54:07.199077 4544 raft_consensus.cc:374] T 00000000000000000000000000000000 P 39ad015d374848d084369d0f182d3f51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:54:07.199098 4544 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39ad015d374848d084369d0f182d3f51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39ad015d374848d084369d0f182d3f51, State: Initialized, Role: FOLLOWER
I20260928 21:54:07.199234 4544 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39ad015d374848d084369d0f182d3f51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39ad015d374848d084369d0f182d3f51" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 44601 } }
I20260928 21:54:07.199308 4544 raft_consensus.cc:388] T 00000000000000000000000000000000 P 39ad015d374848d084369d0f182d3f51 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:54:07.199365 4544 raft_consensus.cc:482] T 00000000000000000000000000000000 P 39ad015d374848d084369d0f182d3f51 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:54:07.199417 4544 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39ad015d374848d084369d0f182d3f51 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:54:07.199662 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 4466
I20260928 21:54:07.199739 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/minicluster-data/master-0/wal/instance
I20260928 21:54:07.199896 2777 external_mini_cluster.cc:1428] Setting key 3df9b44b515bb8753ed1135ab3177156
I20260928 21:54:07.200095 4544 raft_consensus.cc:504] T 00000000000000000000000000000000 P 39ad015d374848d084369d0f182d3f51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39ad015d374848d084369d0f182d3f51" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 44601 } }
I20260928 21:54:07.200207 4544 leader_election.cc:302] T 00000000000000000000000000000000 P 39ad015d374848d084369d0f182d3f51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 39ad015d374848d084369d0f182d3f51; no voters:
I20260928 21:54:07.200388 4544 leader_election.cc:288] T 00000000000000000000000000000000 P 39ad015d374848d084369d0f182d3f51 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:54:07.200611 4548 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 39ad015d374848d084369d0f182d3f51 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:54:07.200666 4544 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39ad015d374848d084369d0f182d3f51 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:54:07.200807 4548 raft_consensus.cc:686] T 00000000000000000000000000000000 P 39ad015d374848d084369d0f182d3f51 [term 1 LEADER]: Becoming Leader. State: Replica: 39ad015d374848d084369d0f182d3f51, State: Running, Role: LEADER
I20260928 21:54:07.200896 4548 consensus_queue.cc:237] T 00000000000000000000000000000000 P 39ad015d374848d084369d0f182d3f51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39ad015d374848d084369d0f182d3f51" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 44601 } }
I20260928 21:54:07.201278 4550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39ad015d374848d084369d0f182d3f51 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "39ad015d374848d084369d0f182d3f51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39ad015d374848d084369d0f182d3f51" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 44601 } } }
I20260928 21:54:07.201362 4550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39ad015d374848d084369d0f182d3f51 [sys.catalog]: This master's current role is: LEADER
I20260928 21:54:07.201517 4551 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39ad015d374848d084369d0f182d3f51 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39ad015d374848d084369d0f182d3f51. Latest consensus state: current_term: 1 leader_uuid: "39ad015d374848d084369d0f182d3f51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39ad015d374848d084369d0f182d3f51" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 44601 } } }
I20260928 21:54:07.201567 4551 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39ad015d374848d084369d0f182d3f51 [sys.catalog]: This master's current role is: LEADER
I20260928 21:54:07.201944 4558 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:54:07.202689 4558 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:54:07.204445 4558 catalog_manager.cc:1347] Generated new cluster ID: 248327815b8c4aba9d08f7d7c422ef17
I20260928 21:54:07.204504 4558 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 21:54:07 dist-test-slave-k6lz krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632447, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:54:07.211650 4549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:07.200357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35178 (local address 127.2.182.126:44601)
0928 21:54:07.200534 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:07.200545 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:07.200579 (+ 34us) server_negotiation.cc:407] Connection header received
0928 21:54:07.200700 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:07.200707 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:07.200768 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:07.200910 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:07.201829 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:07.203299 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:07.204260 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:07.204846 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:07.204947 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:07.207642 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:07.207647 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:07.207647 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:07.207659 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:07.209588 (+ 1929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:07.209963 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:07.209966 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:07.209967 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:07.210077 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:07.210243 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:07.210244 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:07.210244 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:07.210555 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:07.210704 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:07.211205 (+ 501us) server_negotiation.cc:299] Negotiation successful
0928 21:54:07.211395 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":88,"threads_started":1}
I20260928 21:54:07.236941 4558 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:54:07.237600 4558 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:54:07.245973 4558 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 39ad015d374848d084369d0f182d3f51: Generated new TSK 0
I20260928 21:54:07.246188 4558 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.182.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab.
Entry for principal kudu/127.2.182.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab.
I20260928 21:54:07.279573 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.65:0
--local_ip_for_outbound_sockets=127.2.182.65
--webserver_interface=127.2.182.65
--webserver_port=0
--tserver_master_addrs=127.2.182.126:44601
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab
--principal=kudu/127.2.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:33301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:54:07.366505 4570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:54:07.366731 4570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:54:07.366758 4570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:54:07.368042 4570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:54:07.368093 4570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:54:07.368113 4570 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 21:54:07.368124 4570 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 21:54:07.368140 4570 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.65
I20260928 21:54:07.369680 4570 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:33301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab
--principal=kudu/127.2.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.182.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.182.126:44601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4570
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.65
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:54:07.369915 4570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:54:07.370147 4570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:54:07.375494 4570 server_base.cc:1034] running on GCE node
W20260928 21:54:07.375630 4576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:07.375670 4575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:07.375689 4578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:07.376063 4570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:07.376300 4570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:07.377439 4570 hybrid_clock.cc:648] HybridClock initialized: now 1790632447377421 us; error 35 us; skew 500 ppm
Sep 28 21:54:07 dist-test-slave-k6lz krb5kdc[4453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632447, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:54:07.380131 4570 init.cc:377] Logged in from keytab as kudu/127.2.182.65@KRBTEST.COM (short username kudu)
I20260928 21:54:07.380580 4570 webserver.cc:460] Webserver started at http://127.2.182.65:34527/ using document root <none> and password file <none>
I20260928 21:54:07.380810 4570 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:07.380859 4570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:07.380982 4570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:54:07.382071 4570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/minicluster-data/ts-0/data/instance:
uuid: "26ef4abb4a48496e890d727f5e551105"
format_stamp: "Formatted at 2026-09-28 21:54:07 on dist-test-slave-k6lz"
server_key: "3a8dc2b6e51eab79c8f647dbcae55bc0"
server_key_iv: "0941c71934b6535127daa26626d1cc5f"
server_key_version: "encryptionkey@0"
I20260928 21:54:07.382416 4570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/minicluster-data/ts-0/wal/instance:
uuid: "26ef4abb4a48496e890d727f5e551105"
format_stamp: "Formatted at 2026-09-28 21:54:07 on dist-test-slave-k6lz"
server_key: "3a8dc2b6e51eab79c8f647dbcae55bc0"
server_key_iv: "0941c71934b6535127daa26626d1cc5f"
server_key_version: "encryptionkey@0"
I20260928 21:54:07.383695 4570 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 21:54:07.384554 4585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:07.384744 4570 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:54:07.384909 4570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/minicluster-data/ts-0/wal
uuid: "26ef4abb4a48496e890d727f5e551105"
format_stamp: "Formatted at 2026-09-28 21:54:07 on dist-test-slave-k6lz"
server_key: "3a8dc2b6e51eab79c8f647dbcae55bc0"
server_key_iv: "0941c71934b6535127daa26626d1cc5f"
server_key_version: "encryptionkey@0"
I20260928 21:54:07.384979 4570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:07.401487 4570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:54:07.404699 4570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:54:07.404965 4570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:54:07.405594 4570 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:54:07.405649 4570 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:07.405680 4570 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:54:07.405695 4570 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 21:54:07 dist-test-slave-k6lz krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632447, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.65@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:54:07.412923 4570 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.65:43373
I20260928 21:54:07.413087 4700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.65:43373 every 8 connection(s)
I20260928 21:54:07.413333 4570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
I20260928 21:54:07.415472 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 4570
I20260928 21:54:07.415561 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/minicluster-data/ts-0/wal/instance
I20260928 21:54:07.415797 2777 external_mini_cluster.cc:1428] Setting key 10a7e89ccf348153e2dc6df1e0cf71ea
I20260928 21:54:07.416241 4549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:07.406093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.65:43613 (local address 127.2.182.126:44601)
0928 21:54:07.406164 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:07.406167 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:07.406477 (+ 310us) server_negotiation.cc:407] Connection header received
0928 21:54:07.407204 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:07.407206 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:07.407219 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:07.407293 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:07.408227 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:07.408738 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:07.410298 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:07.410548 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:07.410621 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:07.413440 (+ 2819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:07.413442 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:07.413442 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:07.413451 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:07.414864 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:07.415267 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:07.415269 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:07.415269 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:07.415346 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:07.415565 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:07.415566 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:07.415567 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:07.415707 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:07.415765 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:07.416063 (+ 298us) server_negotiation.cc:299] Negotiation successful
0928 21:54:07.416155 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 21:54:07.416428 4594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:07.406069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:44601 (local address 127.2.182.65:43613)
0928 21:54:07.406310 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:07.406362 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:07.407025 (+ 663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:07.407413 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:07.407417 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:07.407809 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:07.408133 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:07.408139 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:07.408809 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:07.408810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:07.410177 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:07.410180 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:07.410640 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:07.410641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:07.410811 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:07.411527 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:07.411539 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:07.413324 (+ 1785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:07.414948 (+ 1624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:07.414955 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:07.414958 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:07.415175 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:07.415396 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:07.415396 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:07.415397 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:07.415509 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:07.415766 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:07.415773 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:07.415933 (+ 160us) client_negotiation.cc:769] Sending connection context
0928 21:54:07.416033 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 21:54:07.416201 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":141,"threads_started":1}
I20260928 21:54:07.416904 4701 heartbeater.cc:347] Connected to a master server at 127.2.182.126:44601
I20260928 21:54:07.417029 4701 heartbeater.cc:464] Registering TS with master...
I20260928 21:54:07.417248 4701 heartbeater.cc:511] Master 127.2.182.126:44601 requested a full tablet report, sending...
I20260928 21:54:07.417848 4498 ts_manager.cc:194] Registered new tserver with Master: 26ef4abb4a48496e890d727f5e551105 (127.2.182.65:43373)
I20260928 21:54:07.418731 4498 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.65@KRBTEST.COM'} at 127.2.182.65:43613
WARNING: no policy specified for kudu/127.2.182.66@KRBTEST.COM; defaulting to no policy
I20260928 21:54:07.424299 4549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:07.417936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.65:49825 (local address 127.2.182.126:44601)
0928 21:54:07.417977 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:07.417979 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:07.418240 (+ 261us) server_negotiation.cc:407] Connection header received
0928 21:54:07.418269 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:07.418269 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:07.418279 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:07.418334 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:07.418660 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:07.419401 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:07.420390 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:07.420576 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:07.420630 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:07.421641 (+ 1011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:07.421642 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:07.421642 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:07.421649 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:07.423010 (+ 1361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:07.423447 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:07.423448 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:07.423449 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:07.423507 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:07.423709 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:07.423710 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:07.423710 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:07.423896 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:07.423952 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:07.424138 (+ 186us) server_negotiation.cc:299] Negotiation successful
0928 21:54:07.424224 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 21:54:07.424499 4708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:07.418007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:44601 (local address 127.2.182.65:49825)
0928 21:54:07.418157 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:07.418168 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:07.418231 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:07.418348 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:07.418348 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:07.418533 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:07.418594 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:07.418596 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:07.419490 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:07.419491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:07.420312 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:07.420316 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:07.420647 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:07.420648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:07.420723 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:07.421130 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:07.421135 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:07.421534 (+ 399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:07.423098 (+ 1564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:07.423099 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:07.423100 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:07.423364 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:07.423572 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:07.423573 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:07.423573 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:07.423646 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:07.423964 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:07.423965 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:07.424022 (+ 57us) client_negotiation.cc:769] Sending connection context
0928 21:54:07.424108 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 21:54:07.424224 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":83,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab.
Entry for principal kudu/127.2.182.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab.
I20260928 21:54:07.443679 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.66:0
--local_ip_for_outbound_sockets=127.2.182.66
--webserver_interface=127.2.182.66
--webserver_port=0
--tserver_master_addrs=127.2.182.126:44601
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab
--principal=kudu/127.2.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:33301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:54:07.525731 4710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:54:07.525969 4710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:54:07.526006 4710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:54:07.527374 4710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:54:07.527434 4710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:54:07.527467 4710 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 21:54:07.527483 4710 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 21:54:07.527513 4710 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.66
I20260928 21:54:07.529110 4710 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:33301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab
--principal=kudu/127.2.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.182.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.182.126:44601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4710
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.66
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:54:07.529537 4710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:54:07.529819 4710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:54:07.535467 4716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:07.535565 4718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:07.535593 4715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:07.535655 4710 server_base.cc:1034] running on GCE node
I20260928 21:54:07.535837 4710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:07.536049 4710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:07.537315 4710 hybrid_clock.cc:648] HybridClock initialized: now 1790632447537263 us; error 64 us; skew 500 ppm
Sep 28 21:54:07 dist-test-slave-k6lz krb5kdc[4453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632447, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:54:07.540102 4710 init.cc:377] Logged in from keytab as kudu/127.2.182.66@KRBTEST.COM (short username kudu)
I20260928 21:54:07.540594 4710 webserver.cc:460] Webserver started at http://127.2.182.66:42511/ using document root <none> and password file <none>
I20260928 21:54:07.540890 4710 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:07.540948 4710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:07.541043 4710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:54:07.542130 4710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/minicluster-data/ts-1/data/instance:
uuid: "abce6d4107764e18b0bccc5d2799434c"
format_stamp: "Formatted at 2026-09-28 21:54:07 on dist-test-slave-k6lz"
server_key: "ded615c63b094c7c348050d01dc09536"
server_key_iv: "b81d19613878e22f1bd2bdb28498aceb"
server_key_version: "encryptionkey@0"
I20260928 21:54:07.542439 4710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/minicluster-data/ts-1/wal/instance:
uuid: "abce6d4107764e18b0bccc5d2799434c"
format_stamp: "Formatted at 2026-09-28 21:54:07 on dist-test-slave-k6lz"
server_key: "ded615c63b094c7c348050d01dc09536"
server_key_iv: "b81d19613878e22f1bd2bdb28498aceb"
server_key_version: "encryptionkey@0"
I20260928 21:54:07.543833 4710 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 21:54:07.544625 4725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:07.544934 4710 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:54:07.545027 4710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/minicluster-data/ts-1/wal
uuid: "abce6d4107764e18b0bccc5d2799434c"
format_stamp: "Formatted at 2026-09-28 21:54:07 on dist-test-slave-k6lz"
server_key: "ded615c63b094c7c348050d01dc09536"
server_key_iv: "b81d19613878e22f1bd2bdb28498aceb"
server_key_version: "encryptionkey@0"
I20260928 21:54:07.545092 4710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:07.555661 4710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:54:07.558002 4710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:54:07.558152 4710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:54:07.558766 4710 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:54:07.558818 4710 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:07.558848 4710 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:54:07.558863 4710 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 21:54:07 dist-test-slave-k6lz krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632447, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.66@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:54:07.565779 4710 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.66:37949
I20260928 21:54:07.565881 4840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.66:37949 every 8 connection(s)
I20260928 21:54:07.566200 4710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
I20260928 21:54:07.568532 4549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:07.559203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.66:37911 (local address 127.2.182.126:44601)
0928 21:54:07.559270 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:07.559273 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:07.559514 (+ 241us) server_negotiation.cc:407] Connection header received
0928 21:54:07.559985 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:07.559987 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:07.560000 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:07.560079 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:07.560740 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:07.561482 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:07.562624 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:07.562936 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:07.563018 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:07.565660 (+ 2642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:07.565662 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:07.565663 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:07.565671 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:07.567164 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:07.567605 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:07.567607 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:07.567607 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:07.567667 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:07.567896 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:07.567897 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:07.567897 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:07.568020 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:07.568075 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:07.568357 (+ 282us) server_negotiation.cc:299] Negotiation successful
0928 21:54:07.568441 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 21:54:07.568674 4734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:07.559207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:44601 (local address 127.2.182.66:37911)
0928 21:54:07.559391 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:07.559431 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:07.559901 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:07.560163 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:07.560167 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:07.560452 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:07.560647 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:07.560653 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:07.561577 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:07.561578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:07.562549 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:07.562552 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:07.563041 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:07.563042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:07.563193 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:07.563853 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:07.563862 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:07.565527 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:07.567251 (+ 1724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:07.567256 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:07.567259 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:07.567529 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:07.567724 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:07.567725 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:07.567725 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:07.567828 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:07.568077 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:07.568083 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:07.568222 (+ 139us) client_negotiation.cc:769] Sending connection context
0928 21:54:07.568304 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 21:54:07.568464 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":84,"threads_started":1}
I20260928 21:54:07.569033 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 4710
I20260928 21:54:07.569185 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/minicluster-data/ts-1/wal/instance
I20260928 21:54:07.569176 4841 heartbeater.cc:347] Connected to a master server at 127.2.182.126:44601
I20260928 21:54:07.569258 4841 heartbeater.cc:464] Registering TS with master...
I20260928 21:54:07.569403 2777 external_mini_cluster.cc:1428] Setting key f4fc3fec112366561eaa7afa37eabf1c
I20260928 21:54:07.569432 4841 heartbeater.cc:511] Master 127.2.182.126:44601 requested a full tablet report, sending...
I20260928 21:54:07.569769 4498 ts_manager.cc:194] Registered new tserver with Master: abce6d4107764e18b0bccc5d2799434c (127.2.182.66:37949)
I20260928 21:54:07.570286 4498 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.66@KRBTEST.COM'} at 127.2.182.66:37911
I20260928 21:54:07.576632 4549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:07.570464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.66:56689 (local address 127.2.182.126:44601)
0928 21:54:07.570500 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:07.570502 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:07.570510 (+ 8us) server_negotiation.cc:407] Connection header received
0928 21:54:07.570567 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:07.570568 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:07.570576 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:07.570622 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:07.571016 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:07.571862 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:07.572830 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:07.573022 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:07.573098 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:07.574087 (+ 989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:07.574089 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:07.574089 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:07.574096 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:07.575541 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:07.575862 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:07.575864 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:07.575864 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:07.575924 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:07.576089 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:07.576090 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:07.576090 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:07.576246 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:07.576304 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:07.576453 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 21:54:07.576532 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 21:54:07.576645 4847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:07.570237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:44601 (local address 127.2.182.66:56689)
0928 21:54:07.570453 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:07.570464 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:07.570514 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:07.570629 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:07.570630 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:07.570831 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:07.570894 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:07.570896 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:07.571985 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:07.571987 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:07.572738 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:07.572741 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:07.573121 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:07.573123 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:07.573213 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:07.573678 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:07.573682 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:07.573988 (+ 306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:07.575628 (+ 1640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:07.575629 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:07.575630 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:07.575799 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:07.575979 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:07.575980 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:07.575980 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:07.576038 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:07.576311 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:07.576312 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:07.576364 (+ 52us) client_negotiation.cc:769] Sending connection context
0928 21:54:07.576422 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 21:54:07.576533 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":104,"threads_started":1}
WARNING: no policy specified for kudu/127.2.182.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab.
Entry for principal kudu/127.2.182.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab.
I20260928 21:54:07.598942 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.67:0
--local_ip_for_outbound_sockets=127.2.182.67
--webserver_interface=127.2.182.67
--webserver_port=0
--tserver_master_addrs=127.2.182.126:44601
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab
--principal=kudu/127.2.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:33301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:54:07.681329 4850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:54:07.681565 4850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:54:07.681593 4850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:54:07.682947 4850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:54:07.683017 4850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:54:07.683039 4850 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 21:54:07.683049 4850 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 21:54:07.683068 4850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.67
I20260928 21:54:07.684736 4850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:33301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab
--principal=kudu/127.2.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.182.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.182.126:44601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.67
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:54:07.684974 4850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:54:07.685166 4850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:54:07.690514 4856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:07.690500 4855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:07.690500 4858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:07.690884 4850 server_base.cc:1034] running on GCE node
I20260928 21:54:07.691082 4850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:07.691313 4850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:07.692505 4850 hybrid_clock.cc:648] HybridClock initialized: now 1790632447692489 us; error 28 us; skew 500 ppm
Sep 28 21:54:07 dist-test-slave-k6lz krb5kdc[4453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632447, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:54:07.695254 4850 init.cc:377] Logged in from keytab as kudu/127.2.182.67@KRBTEST.COM (short username kudu)
I20260928 21:54:07.695770 4850 webserver.cc:460] Webserver started at http://127.2.182.67:41881/ using document root <none> and password file <none>
I20260928 21:54:07.696012 4850 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:07.696063 4850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:07.696189 4850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:54:07.697248 4850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/minicluster-data/ts-2/data/instance:
uuid: "75040f489b5f42d3b5a1e3f264acdd2b"
format_stamp: "Formatted at 2026-09-28 21:54:07 on dist-test-slave-k6lz"
server_key: "65fb727f4be4bf3d440e67f47b03b4c8"
server_key_iv: "2b9adc35a8db268e42afdab245dd7f9d"
server_key_version: "encryptionkey@0"
I20260928 21:54:07.697593 4850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/minicluster-data/ts-2/wal/instance:
uuid: "75040f489b5f42d3b5a1e3f264acdd2b"
format_stamp: "Formatted at 2026-09-28 21:54:07 on dist-test-slave-k6lz"
server_key: "65fb727f4be4bf3d440e67f47b03b4c8"
server_key_iv: "2b9adc35a8db268e42afdab245dd7f9d"
server_key_version: "encryptionkey@0"
I20260928 21:54:07.698998 4850 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:54:07.699932 4865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:07.700130 4850 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 21:54:07.700225 4850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/minicluster-data/ts-2/wal
uuid: "75040f489b5f42d3b5a1e3f264acdd2b"
format_stamp: "Formatted at 2026-09-28 21:54:07 on dist-test-slave-k6lz"
server_key: "65fb727f4be4bf3d440e67f47b03b4c8"
server_key_iv: "2b9adc35a8db268e42afdab245dd7f9d"
server_key_version: "encryptionkey@0"
I20260928 21:54:07.700348 4850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:07.719066 4850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:54:07.721351 4850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:54:07.721498 4850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:54:07.722098 4850 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:54:07.722149 4850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:07.722178 4850 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:54:07.722193 4850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 21:54:07 dist-test-slave-k6lz krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632447, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.67@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:54:07.728984 4980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.67:35509 every 8 connection(s)
I20260928 21:54:07.728919 4850 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.67:35509
I20260928 21:54:07.729723 4850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
I20260928 21:54:07.732067 4549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:07.722510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.67:34671 (local address 127.2.182.126:44601)
0928 21:54:07.722581 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:07.722583 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:07.722813 (+ 230us) server_negotiation.cc:407] Connection header received
0928 21:54:07.723510 (+ 697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:07.723511 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:07.723525 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:07.723595 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:07.724581 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:07.725310 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:07.726797 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:07.727068 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:07.727134 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:07.729337 (+ 2203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:07.729339 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:07.729339 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:07.729346 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:07.730762 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:07.731158 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:07.731159 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:07.731159 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:07.731206 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:07.731471 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:07.731472 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:07.731472 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:07.731576 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:07.731626 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:07.731942 (+ 316us) server_negotiation.cc:299] Negotiation successful
0928 21:54:07.732004 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 21:54:07.732291 4874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:07.722496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:44601 (local address 127.2.182.67:34671)
0928 21:54:07.722670 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:07.722721 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:07.723387 (+ 666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:07.723692 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:07.723696 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:07.724169 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:07.724470 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:07.724478 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:07.725403 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:07.725404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:07.726708 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:07.726711 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:07.727150 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:07.727152 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:07.727285 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:07.727812 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:07.727821 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:07.729240 (+ 1419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:07.730858 (+ 1618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:07.730865 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:07.730866 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:07.731080 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:07.731250 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:07.731251 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:07.731251 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:07.731408 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:07.731691 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:07.731697 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:07.731821 (+ 124us) client_negotiation.cc:769] Sending connection context
0928 21:54:07.731914 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 21:54:07.732048 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":87,"threads_started":1}
I20260928 21:54:07.732738 4981 heartbeater.cc:347] Connected to a master server at 127.2.182.126:44601
I20260928 21:54:07.732818 4981 heartbeater.cc:464] Registering TS with master...
I20260928 21:54:07.733024 4981 heartbeater.cc:511] Master 127.2.182.126:44601 requested a full tablet report, sending...
I20260928 21:54:07.733399 4497 ts_manager.cc:194] Registered new tserver with Master: 75040f489b5f42d3b5a1e3f264acdd2b (127.2.182.67:35509)
I20260928 21:54:07.733889 4497 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.67@KRBTEST.COM'} at 127.2.182.67:34671
I20260928 21:54:07.734064 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 4850
I20260928 21:54:07.734144 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/minicluster-data/ts-2/wal/instance
I20260928 21:54:07.734319 2777 external_mini_cluster.cc:1428] Setting key 4fd1585561ce95176e244dde51299ee2
I20260928 21:54:07.735251 2777 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:54:07.740434 4987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:07.733861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:44601 (local address 127.2.182.67:45627)
0928 21:54:07.734473 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:07.734485 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:07.734548 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:07.734739 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:07.734739 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:07.734913 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:07.734966 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:07.734968 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:07.735828 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:07.735829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:07.736328 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:07.736329 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:07.736766 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:07.736767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:07.736916 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:07.737338 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:07.737343 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:07.737729 (+ 386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:07.739358 (+ 1629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:07.739359 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:07.739359 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:07.739569 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:07.739842 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:07.739843 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:07.739843 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:07.739898 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:07.740183 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:07.740183 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:07.740223 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 21:54:07.740272 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 21:54:07.740363 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":70,"threads_started":1}
I20260928 21:54:07.740469 4549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:07.733954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.67:45627 (local address 127.2.182.126:44601)
0928 21:54:07.733995 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:07.733996 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:07.734534 (+ 538us) server_negotiation.cc:407] Connection header received
0928 21:54:07.734593 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:07.734594 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:07.734604 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:07.734717 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:07.735027 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:07.735741 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:07.736407 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:07.736671 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:07.736736 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:07.737831 (+ 1095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:07.737832 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:07.737833 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:07.737839 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:07.739248 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:07.739658 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:07.739660 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:07.739660 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:07.739728 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:07.739958 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:07.739959 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:07.739959 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:07.740112 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:07.740179 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:07.740308 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 21:54:07.740382 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
Sep 28 21:54:07 dist-test-slave-k6lz krb5kdc[4453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632447, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 21:54:07 dist-test-slave-k6lz krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632447, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:54:07.758514 4549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:07.751372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42846 (local address 127.2.182.126:44601)
0928 21:54:07.751416 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:07.751418 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:07.751426 (+ 8us) server_negotiation.cc:407] Connection header received
0928 21:54:07.751481 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:07.751481 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:07.751491 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:07.751537 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:07.752127 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:07.752620 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:07.753406 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:07.753600 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:07.753652 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:07.756013 (+ 2361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:07.756014 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:07.756015 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:07.756022 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:07.757378 (+ 1356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:07.757782 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:07.757783 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:07.757783 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:07.757837 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:07.758048 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:07.758048 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:07.758049 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:07.758208 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:07.758284 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:07.758371 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 21:54:07.758440 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 21:54:07.760361 4497 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42846:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:54:07.762077 4497 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:54:07.768951 4999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:07.764583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.67:35509 (local address 127.0.0.1:38864)
0928 21:54:07.764749 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:07.764791 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:07.764856 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:07.765140 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:07.765144 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:07.765146 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:54:07.765218 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:07.765220 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:07.766542 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:07.766543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:07.767873 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:07.767876 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:07.768613 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:07.768614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:07.768683 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:07.768687 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:54:07.768734 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 21:54:07.768778 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":73,"thread_start_us":48,"threads_started":1}
I20260928 21:54:07.768953 4998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:07.764881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.66:37949 (local address 127.0.0.1:44464)
0928 21:54:07.765248 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:07.765258 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:07.765298 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:07.765524 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:07.765525 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:07.765526 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:54:07.765598 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:07.765599 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:07.766471 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:07.766472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:07.767875 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:07.767876 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:07.768720 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:07.768721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:07.768780 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:07.768784 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:54:07.768828 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 21:54:07.768866 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"mutex_wait_us":93,"thread_start_us":82,"threads_started":1}
I20260928 21:54:07.769170 5001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:07.764782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38864 (local address 127.2.182.67:35509)
0928 21:54:07.764945 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:07.764950 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:07.764962 (+ 12us) server_negotiation.cc:407] Connection header received
0928 21:54:07.765001 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:07.765003 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:07.765018 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:07.765132 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:54:07.765302 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:07.766471 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:07.767971 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:07.768487 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:07.768579 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:07.768636 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:07.768843 (+ 207us) server_negotiation.cc:299] Negotiation successful
0928 21:54:07.768918 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":69,"threads_started":1}
I20260928 21:54:07.769282 5000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:07.764765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33448 (local address 127.2.182.65:43373)
0928 21:54:07.764947 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:07.764951 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:07.765064 (+ 113us) server_negotiation.cc:407] Connection header received
0928 21:54:07.765112 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:07.765115 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:07.765128 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:07.765193 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:54:07.765470 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:07.766509 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:07.767718 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:07.768390 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:07.768466 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:07.768527 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:07.769153 (+ 626us) server_negotiation.cc:299] Negotiation successful
0928 21:54:07.769190 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":87,"threads_started":1}
I20260928 21:54:07.768795 5002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:07.764588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.65:43373 (local address 127.0.0.1:33448)
0928 21:54:07.764996 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:07.765005 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:07.765053 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:07.765350 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:07.765351 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:07.765353 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:54:07.765402 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:07.765404 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:07.766587 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:07.766588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:07.767643 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:07.767646 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:07.768480 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:07.768481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:07.768563 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:07.768579 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 21:54:07.768641 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 21:54:07.768694 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"mutex_wait_us":60,"thread_start_us":106,"threads_started":1}
I20260928 21:54:07.770123 4628 tablet_service.cc:1467] Processing CreateTablet for tablet 5548267a9c6446fab81daad93c3a00fb (DEFAULT_TABLE table=test-table [id=9e27fe05f25d49fd969310fd488d5172]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:54:07.770123 4915 tablet_service.cc:1467] Processing CreateTablet for tablet 5548267a9c6446fab81daad93c3a00fb (DEFAULT_TABLE table=test-table [id=9e27fe05f25d49fd969310fd488d5172]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:54:07.770493 4628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5548267a9c6446fab81daad93c3a00fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:54:07.770493 4915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5548267a9c6446fab81daad93c3a00fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:54:07.768982 5003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:07.765094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44464 (local address 127.2.182.66:37949)
0928 21:54:07.765323 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:07.765328 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:07.765338 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:54:07.765379 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:07.765381 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:07.765395 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:07.765463 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:54:07.765663 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:07.766389 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:07.767960 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:07.768653 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:07.768719 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:07.768772 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:07.768850 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 21:54:07.768889 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":67,"threads_started":1}
I20260928 21:54:07.771989 4775 tablet_service.cc:1467] Processing CreateTablet for tablet 5548267a9c6446fab81daad93c3a00fb (DEFAULT_TABLE table=test-table [id=9e27fe05f25d49fd969310fd488d5172]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:54:07.772320 4775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5548267a9c6446fab81daad93c3a00fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:54:07.773883 5004 tablet_bootstrap.cc:492] T 5548267a9c6446fab81daad93c3a00fb P 75040f489b5f42d3b5a1e3f264acdd2b: Bootstrap starting.
I20260928 21:54:07.774267 5005 tablet_bootstrap.cc:492] T 5548267a9c6446fab81daad93c3a00fb P 26ef4abb4a48496e890d727f5e551105: Bootstrap starting.
I20260928 21:54:07.774791 5005 tablet_bootstrap.cc:654] T 5548267a9c6446fab81daad93c3a00fb P 26ef4abb4a48496e890d727f5e551105: Neither blocks nor log segments found. Creating new log.
I20260928 21:54:07.774947 5004 tablet_bootstrap.cc:654] T 5548267a9c6446fab81daad93c3a00fb P 75040f489b5f42d3b5a1e3f264acdd2b: Neither blocks nor log segments found. Creating new log.
I20260928 21:54:07.775115 5005 log.cc:824] T 5548267a9c6446fab81daad93c3a00fb P 26ef4abb4a48496e890d727f5e551105: Log is configured to *not* fsync() on all Append() calls
I20260928 21:54:07.775269 5004 log.cc:824] T 5548267a9c6446fab81daad93c3a00fb P 75040f489b5f42d3b5a1e3f264acdd2b: Log is configured to *not* fsync() on all Append() calls
I20260928 21:54:07.775595 5007 tablet_bootstrap.cc:492] T 5548267a9c6446fab81daad93c3a00fb P abce6d4107764e18b0bccc5d2799434c: Bootstrap starting.
I20260928 21:54:07.775913 5005 tablet_bootstrap.cc:492] T 5548267a9c6446fab81daad93c3a00fb P 26ef4abb4a48496e890d727f5e551105: No bootstrap required, opened a new log
I20260928 21:54:07.775979 5005 ts_tablet_manager.cc:1397] T 5548267a9c6446fab81daad93c3a00fb P 26ef4abb4a48496e890d727f5e551105: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:54:07.776170 5007 tablet_bootstrap.cc:654] T 5548267a9c6446fab81daad93c3a00fb P abce6d4107764e18b0bccc5d2799434c: Neither blocks nor log segments found. Creating new log.
I20260928 21:54:07.776434 5007 log.cc:824] T 5548267a9c6446fab81daad93c3a00fb P abce6d4107764e18b0bccc5d2799434c: Log is configured to *not* fsync() on all Append() calls
I20260928 21:54:07.777052 5004 tablet_bootstrap.cc:492] T 5548267a9c6446fab81daad93c3a00fb P 75040f489b5f42d3b5a1e3f264acdd2b: No bootstrap required, opened a new log
I20260928 21:54:07.777118 5004 ts_tablet_manager.cc:1397] T 5548267a9c6446fab81daad93c3a00fb P 75040f489b5f42d3b5a1e3f264acdd2b: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260928 21:54:07.777135 5007 tablet_bootstrap.cc:492] T 5548267a9c6446fab81daad93c3a00fb P abce6d4107764e18b0bccc5d2799434c: No bootstrap required, opened a new log
I20260928 21:54:07.777199 5007 ts_tablet_manager.cc:1397] T 5548267a9c6446fab81daad93c3a00fb P abce6d4107764e18b0bccc5d2799434c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:54:07.777760 5005 raft_consensus.cc:348] T 5548267a9c6446fab81daad93c3a00fb P 26ef4abb4a48496e890d727f5e551105 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75040f489b5f42d3b5a1e3f264acdd2b" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 35509 } } peers { permanent_uuid: "26ef4abb4a48496e890d727f5e551105" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 43373 } } peers { permanent_uuid: "abce6d4107764e18b0bccc5d2799434c" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37949 } }
I20260928 21:54:07.777932 5005 raft_consensus.cc:374] T 5548267a9c6446fab81daad93c3a00fb P 26ef4abb4a48496e890d727f5e551105 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:54:07.777967 5005 raft_consensus.cc:729] T 5548267a9c6446fab81daad93c3a00fb P 26ef4abb4a48496e890d727f5e551105 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26ef4abb4a48496e890d727f5e551105, State: Initialized, Role: FOLLOWER
I20260928 21:54:07.778083 5005 consensus_queue.cc:260] T 5548267a9c6446fab81daad93c3a00fb P 26ef4abb4a48496e890d727f5e551105 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75040f489b5f42d3b5a1e3f264acdd2b" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 35509 } } peers { permanent_uuid: "26ef4abb4a48496e890d727f5e551105" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 43373 } } peers { permanent_uuid: "abce6d4107764e18b0bccc5d2799434c" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37949 } }
I20260928 21:54:07.778362 5005 ts_tablet_manager.cc:1428] T 5548267a9c6446fab81daad93c3a00fb P 26ef4abb4a48496e890d727f5e551105: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 21:54:07.778370 4701 heartbeater.cc:503] Master 127.2.182.126:44601 was elected leader, sending a full tablet report...
I20260928 21:54:07.778769 5004 raft_consensus.cc:348] T 5548267a9c6446fab81daad93c3a00fb P 75040f489b5f42d3b5a1e3f264acdd2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75040f489b5f42d3b5a1e3f264acdd2b" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 35509 } } peers { permanent_uuid: "26ef4abb4a48496e890d727f5e551105" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 43373 } } peers { permanent_uuid: "abce6d4107764e18b0bccc5d2799434c" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37949 } }
I20260928 21:54:07.778918 5004 raft_consensus.cc:374] T 5548267a9c6446fab81daad93c3a00fb P 75040f489b5f42d3b5a1e3f264acdd2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:54:07.778949 5004 raft_consensus.cc:729] T 5548267a9c6446fab81daad93c3a00fb P 75040f489b5f42d3b5a1e3f264acdd2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75040f489b5f42d3b5a1e3f264acdd2b, State: Initialized, Role: FOLLOWER
I20260928 21:54:07.778900 5007 raft_consensus.cc:348] T 5548267a9c6446fab81daad93c3a00fb P abce6d4107764e18b0bccc5d2799434c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75040f489b5f42d3b5a1e3f264acdd2b" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 35509 } } peers { permanent_uuid: "26ef4abb4a48496e890d727f5e551105" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 43373 } } peers { permanent_uuid: "abce6d4107764e18b0bccc5d2799434c" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37949 } }
I20260928 21:54:07.779006 5007 raft_consensus.cc:374] T 5548267a9c6446fab81daad93c3a00fb P abce6d4107764e18b0bccc5d2799434c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:54:07.779026 5007 raft_consensus.cc:729] T 5548267a9c6446fab81daad93c3a00fb P abce6d4107764e18b0bccc5d2799434c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abce6d4107764e18b0bccc5d2799434c, State: Initialized, Role: FOLLOWER
I20260928 21:54:07.779049 5004 consensus_queue.cc:260] T 5548267a9c6446fab81daad93c3a00fb P 75040f489b5f42d3b5a1e3f264acdd2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75040f489b5f42d3b5a1e3f264acdd2b" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 35509 } } peers { permanent_uuid: "26ef4abb4a48496e890d727f5e551105" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 43373 } } peers { permanent_uuid: "abce6d4107764e18b0bccc5d2799434c" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37949 } }
I20260928 21:54:07.779121 5007 consensus_queue.cc:260] T 5548267a9c6446fab81daad93c3a00fb P abce6d4107764e18b0bccc5d2799434c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75040f489b5f42d3b5a1e3f264acdd2b" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 35509 } } peers { permanent_uuid: "26ef4abb4a48496e890d727f5e551105" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 43373 } } peers { permanent_uuid: "abce6d4107764e18b0bccc5d2799434c" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37949 } }
I20260928 21:54:07.779312 5004 ts_tablet_manager.cc:1428] T 5548267a9c6446fab81daad93c3a00fb P 75040f489b5f42d3b5a1e3f264acdd2b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 21:54:07.779386 4981 heartbeater.cc:503] Master 127.2.182.126:44601 was elected leader, sending a full tablet report...
I20260928 21:54:07.779382 5007 ts_tablet_manager.cc:1428] T 5548267a9c6446fab81daad93c3a00fb P abce6d4107764e18b0bccc5d2799434c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 21:54:07.779515 4841 heartbeater.cc:503] Master 127.2.182.126:44601 was elected leader, sending a full tablet report...
W20260928 21:54:07.817021 4842 tablet.cc:2369] T 5548267a9c6446fab81daad93c3a00fb P abce6d4107764e18b0bccc5d2799434c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 21:54:07.914394 4702 tablet.cc:2369] T 5548267a9c6446fab81daad93c3a00fb P 26ef4abb4a48496e890d727f5e551105: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:54:07.932624 5012 raft_consensus.cc:482] T 5548267a9c6446fab81daad93c3a00fb P abce6d4107764e18b0bccc5d2799434c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:54:07.932770 5012 raft_consensus.cc:504] T 5548267a9c6446fab81daad93c3a00fb P abce6d4107764e18b0bccc5d2799434c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75040f489b5f42d3b5a1e3f264acdd2b" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 35509 } } peers { permanent_uuid: "26ef4abb4a48496e890d727f5e551105" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 43373 } } peers { permanent_uuid: "abce6d4107764e18b0bccc5d2799434c" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37949 } }
I20260928 21:54:07.933145 5012 leader_election.cc:288] T 5548267a9c6446fab81daad93c3a00fb P abce6d4107764e18b0bccc5d2799434c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75040f489b5f42d3b5a1e3f264acdd2b (127.2.182.67:35509), 26ef4abb4a48496e890d727f5e551105 (127.2.182.65:43373)
I20260928 21:54:07.935993 4734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:07.933360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.65:43373 (local address 127.2.182.66:50147)
0928 21:54:07.933410 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:07.933430 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:07.933481 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:07.933642 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:07.933643 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:07.933645 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:54:07.933714 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:07.933716 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:07.934350 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:07.934351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:07.935081 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:07.935083 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:07.935757 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:07.935758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:07.935828 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:07.935832 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:54:07.935884 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 21:54:07.935917 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 21:54:07.936009 5000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:07.933416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.66:50147 (local address 127.2.182.65:43373)
0928 21:54:07.933483 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:07.933485 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:07.933512 (+ 27us) server_negotiation.cc:407] Connection header received
0928 21:54:07.933538 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:07.933539 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:07.933547 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:07.933625 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:54:07.933774 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:07.934283 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:07.935177 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:07.935686 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:07.935761 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:07.935769 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:07.935898 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 21:54:07.935938 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 21:54:07.936225 5001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:07.933420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.66:35215 (local address 127.2.182.67:35509)
0928 21:54:07.933487 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:07.933489 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:07.933747 (+ 258us) server_negotiation.cc:407] Connection header received
0928 21:54:07.933793 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:07.933794 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:07.933803 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:07.933835 (+ 32us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:54:07.934047 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:07.934532 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:07.935404 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:07.935894 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:07.935956 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:07.935963 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:07.936122 (+ 159us) server_negotiation.cc:299] Negotiation successful
0928 21:54:07.936167 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 21:54:07.936332 4655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5548267a9c6446fab81daad93c3a00fb" candidate_uuid: "abce6d4107764e18b0bccc5d2799434c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26ef4abb4a48496e890d727f5e551105" is_pre_election: true
I20260928 21:54:07.936429 5013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:07.933360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.67:35509 (local address 127.2.182.66:35215)
0928 21:54:07.933689 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:07.933696 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:07.933740 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:07.933924 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:07.933924 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:07.933926 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:54:07.933991 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:07.933992 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:07.934604 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:07.934605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:07.935317 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:07.935319 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:07.935968 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:07.935968 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:07.936035 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:07.936039 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:54:07.936304 (+ 265us) client_negotiation.cc:240] Negotiation successful
0928 21:54:07.936348 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":153,"thread_start_us":99,"threads_started":1}
I20260928 21:54:07.936496 4655 raft_consensus.cc:2454] T 5548267a9c6446fab81daad93c3a00fb P 26ef4abb4a48496e890d727f5e551105 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abce6d4107764e18b0bccc5d2799434c in term 0.
I20260928 21:54:07.936677 4935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5548267a9c6446fab81daad93c3a00fb" candidate_uuid: "abce6d4107764e18b0bccc5d2799434c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75040f489b5f42d3b5a1e3f264acdd2b" is_pre_election: true
I20260928 21:54:07.936726 4728 leader_election.cc:302] T 5548267a9c6446fab81daad93c3a00fb P abce6d4107764e18b0bccc5d2799434c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26ef4abb4a48496e890d727f5e551105, abce6d4107764e18b0bccc5d2799434c; no voters:
I20260928 21:54:07.936785 4935 raft_consensus.cc:2454] T 5548267a9c6446fab81daad93c3a00fb P 75040f489b5f42d3b5a1e3f264acdd2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abce6d4107764e18b0bccc5d2799434c in term 0.
I20260928 21:54:07.936888 5012 raft_consensus.cc:2781] T 5548267a9c6446fab81daad93c3a00fb P abce6d4107764e18b0bccc5d2799434c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:54:07.936946 5012 raft_consensus.cc:482] T 5548267a9c6446fab81daad93c3a00fb P abce6d4107764e18b0bccc5d2799434c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:54:07.936962 5012 raft_consensus.cc:3037] T 5548267a9c6446fab81daad93c3a00fb P abce6d4107764e18b0bccc5d2799434c [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:54:07.937880 5012 raft_consensus.cc:504] T 5548267a9c6446fab81daad93c3a00fb P abce6d4107764e18b0bccc5d2799434c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75040f489b5f42d3b5a1e3f264acdd2b" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 35509 } } peers { permanent_uuid: "26ef4abb4a48496e890d727f5e551105" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 43373 } } peers { permanent_uuid: "abce6d4107764e18b0bccc5d2799434c" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37949 } }
I20260928 21:54:07.937999 5012 leader_election.cc:288] T 5548267a9c6446fab81daad93c3a00fb P abce6d4107764e18b0bccc5d2799434c [CANDIDATE]: Term 1 election: Requested vote from peers 75040f489b5f42d3b5a1e3f264acdd2b (127.2.182.67:35509), 26ef4abb4a48496e890d727f5e551105 (127.2.182.65:43373)
I20260928 21:54:07.938200 4655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5548267a9c6446fab81daad93c3a00fb" candidate_uuid: "abce6d4107764e18b0bccc5d2799434c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26ef4abb4a48496e890d727f5e551105"
I20260928 21:54:07.938213 4935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5548267a9c6446fab81daad93c3a00fb" candidate_uuid: "abce6d4107764e18b0bccc5d2799434c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75040f489b5f42d3b5a1e3f264acdd2b"
I20260928 21:54:07.938285 4935 raft_consensus.cc:3037] T 5548267a9c6446fab81daad93c3a00fb P 75040f489b5f42d3b5a1e3f264acdd2b [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:54:07.938290 4655 raft_consensus.cc:3037] T 5548267a9c6446fab81daad93c3a00fb P 26ef4abb4a48496e890d727f5e551105 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:54:07.938926 4935 raft_consensus.cc:2454] T 5548267a9c6446fab81daad93c3a00fb P 75040f489b5f42d3b5a1e3f264acdd2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abce6d4107764e18b0bccc5d2799434c in term 1.
I20260928 21:54:07.938995 4655 raft_consensus.cc:2454] T 5548267a9c6446fab81daad93c3a00fb P 26ef4abb4a48496e890d727f5e551105 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abce6d4107764e18b0bccc5d2799434c in term 1.
I20260928 21:54:07.939090 4729 leader_election.cc:302] T 5548267a9c6446fab81daad93c3a00fb P abce6d4107764e18b0bccc5d2799434c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75040f489b5f42d3b5a1e3f264acdd2b, abce6d4107764e18b0bccc5d2799434c; no voters:
I20260928 21:54:07.939177 5012 raft_consensus.cc:2781] T 5548267a9c6446fab81daad93c3a00fb P abce6d4107764e18b0bccc5d2799434c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:54:07.939323 5012 raft_consensus.cc:686] T 5548267a9c6446fab81daad93c3a00fb P abce6d4107764e18b0bccc5d2799434c [term 1 LEADER]: Becoming Leader. State: Replica: abce6d4107764e18b0bccc5d2799434c, State: Running, Role: LEADER
I20260928 21:54:07.939462 5012 consensus_queue.cc:237] T 5548267a9c6446fab81daad93c3a00fb P abce6d4107764e18b0bccc5d2799434c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75040f489b5f42d3b5a1e3f264acdd2b" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 35509 } } peers { permanent_uuid: "26ef4abb4a48496e890d727f5e551105" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 43373 } } peers { permanent_uuid: "abce6d4107764e18b0bccc5d2799434c" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37949 } }
I20260928 21:54:07.940349 4497 catalog_manager.cc:5495] T 5548267a9c6446fab81daad93c3a00fb P abce6d4107764e18b0bccc5d2799434c reported cstate change: term changed from 0 to 1, leader changed from <none> to abce6d4107764e18b0bccc5d2799434c (127.2.182.66). New cstate: current_term: 1 leader_uuid: "abce6d4107764e18b0bccc5d2799434c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75040f489b5f42d3b5a1e3f264acdd2b" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 35509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26ef4abb4a48496e890d727f5e551105" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 43373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "abce6d4107764e18b0bccc5d2799434c" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37949 } health_report { overall_health: HEALTHY } } }
Sep 28 21:54:07 dist-test-slave-k6lz krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632447, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.126@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:54:07.975988 5021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:07.968157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:44601 (local address 127.0.0.1:42848)
0928 21:54:07.968353 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:07.968364 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:07.968438 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:07.968604 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:07.968605 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:07.968963 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:07.969019 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:07.969021 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:07.969929 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:07.969930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:07.970554 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:07.970556 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:07.970995 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:07.970996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:07.971049 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:07.971649 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:07.971654 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:07.973111 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:07.974815 (+ 1704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:07.974825 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:07.974828 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:07.975039 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:07.975255 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:07.975256 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:07.975256 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:07.975323 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:07.975640 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:07.975648 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:07.975736 (+ 88us) client_negotiation.cc:769] Sending connection context
0928 21:54:07.975803 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 21:54:07.975911 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":101,"threads_started":1}
I20260928 21:54:07.976053 4549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:07.968203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42848 (local address 127.2.182.126:44601)
0928 21:54:07.968302 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:07.968304 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:07.968451 (+ 147us) server_negotiation.cc:407] Connection header received
0928 21:54:07.968479 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:07.968480 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:07.968507 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:07.968559 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:07.969129 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:07.969752 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:07.970662 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:07.970905 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:07.970971 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:07.973205 (+ 2234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:07.973207 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:07.973207 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:07.973214 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:07.974715 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:07.975121 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:07.975122 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:07.975122 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:07.975177 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:07.975422 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:07.975423 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:07.975423 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:07.975535 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:07.975584 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:07.975865 (+ 281us) server_negotiation.cc:299] Negotiation successful
0928 21:54:07.975966 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":66}
I20260928 21:54:07.978292 4497 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.2.182.126@KRBTEST.COM'} at 127.0.0.1:42848:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 21:54:07.978492 4497 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:54:07.980219 4775 tablet_service.cc:1467] Processing CreateTablet for tablet 3a7d6d6b71494ecc9d1e5da480be64a5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=305dc819e5f7468eb650fec5722ce13b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 21:54:07.980189 4628 tablet_service.cc:1467] Processing CreateTablet for tablet 3a7d6d6b71494ecc9d1e5da480be64a5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=305dc819e5f7468eb650fec5722ce13b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 21:54:07.980213 4915 tablet_service.cc:1467] Processing CreateTablet for tablet 3a7d6d6b71494ecc9d1e5da480be64a5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=305dc819e5f7468eb650fec5722ce13b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 21:54:07.980384 4775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a7d6d6b71494ecc9d1e5da480be64a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:54:07.980386 4628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a7d6d6b71494ecc9d1e5da480be64a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:54:07.980393 4915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a7d6d6b71494ecc9d1e5da480be64a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:54:07.981644 5007 tablet_bootstrap.cc:492] T 3a7d6d6b71494ecc9d1e5da480be64a5 P abce6d4107764e18b0bccc5d2799434c: Bootstrap starting.
W20260928 21:54:07.981741 4982 tablet.cc:2369] T 5548267a9c6446fab81daad93c3a00fb P 75040f489b5f42d3b5a1e3f264acdd2b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:54:07.981904 5004 tablet_bootstrap.cc:492] T 3a7d6d6b71494ecc9d1e5da480be64a5 P 75040f489b5f42d3b5a1e3f264acdd2b: Bootstrap starting.
I20260928 21:54:07.982070 5007 tablet_bootstrap.cc:654] T 3a7d6d6b71494ecc9d1e5da480be64a5 P abce6d4107764e18b0bccc5d2799434c: Neither blocks nor log segments found. Creating new log.
I20260928 21:54:07.982362 5004 tablet_bootstrap.cc:654] T 3a7d6d6b71494ecc9d1e5da480be64a5 P 75040f489b5f42d3b5a1e3f264acdd2b: Neither blocks nor log segments found. Creating new log.
I20260928 21:54:07.982687 5005 tablet_bootstrap.cc:492] T 3a7d6d6b71494ecc9d1e5da480be64a5 P 26ef4abb4a48496e890d727f5e551105: Bootstrap starting.
I20260928 21:54:07.982720 5007 tablet_bootstrap.cc:492] T 3a7d6d6b71494ecc9d1e5da480be64a5 P abce6d4107764e18b0bccc5d2799434c: No bootstrap required, opened a new log
I20260928 21:54:07.982761 5007 ts_tablet_manager.cc:1397] T 3a7d6d6b71494ecc9d1e5da480be64a5 P abce6d4107764e18b0bccc5d2799434c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:54:07.982905 5007 raft_consensus.cc:348] T 3a7d6d6b71494ecc9d1e5da480be64a5 P abce6d4107764e18b0bccc5d2799434c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26ef4abb4a48496e890d727f5e551105" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 43373 } } peers { permanent_uuid: "abce6d4107764e18b0bccc5d2799434c" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37949 } } peers { permanent_uuid: "75040f489b5f42d3b5a1e3f264acdd2b" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 35509 } }
I20260928 21:54:07.982988 5007 raft_consensus.cc:374] T 3a7d6d6b71494ecc9d1e5da480be64a5 P abce6d4107764e18b0bccc5d2799434c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:54:07.983001 5007 raft_consensus.cc:729] T 3a7d6d6b71494ecc9d1e5da480be64a5 P abce6d4107764e18b0bccc5d2799434c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abce6d4107764e18b0bccc5d2799434c, State: Initialized, Role: FOLLOWER
I20260928 21:54:07.983055 5005 tablet_bootstrap.cc:654] T 3a7d6d6b71494ecc9d1e5da480be64a5 P 26ef4abb4a48496e890d727f5e551105: Neither blocks nor log segments found. Creating new log.
I20260928 21:54:07.983043 5007 consensus_queue.cc:260] T 3a7d6d6b71494ecc9d1e5da480be64a5 P abce6d4107764e18b0bccc5d2799434c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26ef4abb4a48496e890d727f5e551105" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 43373 } } peers { permanent_uuid: "abce6d4107764e18b0bccc5d2799434c" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37949 } } peers { permanent_uuid: "75040f489b5f42d3b5a1e3f264acdd2b" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 35509 } }
I20260928 21:54:07.983140 5007 ts_tablet_manager.cc:1428] T 3a7d6d6b71494ecc9d1e5da480be64a5 P abce6d4107764e18b0bccc5d2799434c: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 21:54:07.983207 5012 tablet_replica.cc:439] T 3a7d6d6b71494ecc9d1e5da480be64a5 P abce6d4107764e18b0bccc5d2799434c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26ef4abb4a48496e890d727f5e551105" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 43373 } } peers { permanent_uuid: "abce6d4107764e18b0bccc5d2799434c" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37949 } } peers { permanent_uuid: "75040f489b5f42d3b5a1e3f264acdd2b" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 35509 } } }
I20260928 21:54:07.983304 5012 tablet_replica.cc:442] T 3a7d6d6b71494ecc9d1e5da480be64a5 P abce6d4107764e18b0bccc5d2799434c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:54:07.983615 5004 tablet_bootstrap.cc:492] T 3a7d6d6b71494ecc9d1e5da480be64a5 P 75040f489b5f42d3b5a1e3f264acdd2b: No bootstrap required, opened a new log
I20260928 21:54:07.983668 5004 ts_tablet_manager.cc:1397] T 3a7d6d6b71494ecc9d1e5da480be64a5 P 75040f489b5f42d3b5a1e3f264acdd2b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 21:54:07.983821 5004 raft_consensus.cc:348] T 3a7d6d6b71494ecc9d1e5da480be64a5 P 75040f489b5f42d3b5a1e3f264acdd2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26ef4abb4a48496e890d727f5e551105" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 43373 } } peers { permanent_uuid: "abce6d4107764e18b0bccc5d2799434c" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37949 } } peers { permanent_uuid: "75040f489b5f42d3b5a1e3f264acdd2b" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 35509 } }
I20260928 21:54:07.983880 5004 raft_consensus.cc:374] T 3a7d6d6b71494ecc9d1e5da480be64a5 P 75040f489b5f42d3b5a1e3f264acdd2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:54:07.983898 5004 raft_consensus.cc:729] T 3a7d6d6b71494ecc9d1e5da480be64a5 P 75040f489b5f42d3b5a1e3f264acdd2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75040f489b5f42d3b5a1e3f264acdd2b, State: Initialized, Role: FOLLOWER
I20260928 21:54:07.983938 5004 consensus_queue.cc:260] T 3a7d6d6b71494ecc9d1e5da480be64a5 P 75040f489b5f42d3b5a1e3f264acdd2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26ef4abb4a48496e890d727f5e551105" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 43373 } } peers { permanent_uuid: "abce6d4107764e18b0bccc5d2799434c" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37949 } } peers { permanent_uuid: "75040f489b5f42d3b5a1e3f264acdd2b" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 35509 } }
I20260928 21:54:07.984022 5004 ts_tablet_manager.cc:1428] T 3a7d6d6b71494ecc9d1e5da480be64a5 P 75040f489b5f42d3b5a1e3f264acdd2b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:07.984102 5011 tablet_replica.cc:439] T 3a7d6d6b71494ecc9d1e5da480be64a5 P 75040f489b5f42d3b5a1e3f264acdd2b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26ef4abb4a48496e890d727f5e551105" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 43373 } } peers { permanent_uuid: "abce6d4107764e18b0bccc5d2799434c" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37949 } } peers { permanent_uuid: "75040f489b5f42d3b5a1e3f264acdd2b" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 35509 } } }
I20260928 21:54:07.984223 5005 tablet_bootstrap.cc:492] T 3a7d6d6b71494ecc9d1e5da480be64a5 P 26ef4abb4a48496e890d727f5e551105: No bootstrap required, opened a new log
I20260928 21:54:07.984228 5011 tablet_replica.cc:442] T 3a7d6d6b71494ecc9d1e5da480be64a5 P 75040f489b5f42d3b5a1e3f264acdd2b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:54:07.984269 5005 ts_tablet_manager.cc:1397] T 3a7d6d6b71494ecc9d1e5da480be64a5 P 26ef4abb4a48496e890d727f5e551105: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:54:07.984406 5005 raft_consensus.cc:348] T 3a7d6d6b71494ecc9d1e5da480be64a5 P 26ef4abb4a48496e890d727f5e551105 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26ef4abb4a48496e890d727f5e551105" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 43373 } } peers { permanent_uuid: "abce6d4107764e18b0bccc5d2799434c" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37949 } } peers { permanent_uuid: "75040f489b5f42d3b5a1e3f264acdd2b" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 35509 } }
I20260928 21:54:07.984473 5005 raft_consensus.cc:374] T 3a7d6d6b71494ecc9d1e5da480be64a5 P 26ef4abb4a48496e890d727f5e551105 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:54:07.984493 5005 raft_consensus.cc:729] T 3a7d6d6b71494ecc9d1e5da480be64a5 P 26ef4abb4a48496e890d727f5e551105 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26ef4abb4a48496e890d727f5e551105, State: Initialized, Role: FOLLOWER
I20260928 21:54:07.984539 5005 consensus_queue.cc:260] T 3a7d6d6b71494ecc9d1e5da480be64a5 P 26ef4abb4a48496e890d727f5e551105 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26ef4abb4a48496e890d727f5e551105" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 43373 } } peers { permanent_uuid: "abce6d4107764e18b0bccc5d2799434c" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37949 } } peers { permanent_uuid: "75040f489b5f42d3b5a1e3f264acdd2b" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 35509 } }
I20260928 21:54:07.984627 5005 ts_tablet_manager.cc:1428] T 3a7d6d6b71494ecc9d1e5da480be64a5 P 26ef4abb4a48496e890d727f5e551105: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:07.984702 5010 tablet_replica.cc:439] T 3a7d6d6b71494ecc9d1e5da480be64a5 P 26ef4abb4a48496e890d727f5e551105: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26ef4abb4a48496e890d727f5e551105" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 43373 } } peers { permanent_uuid: "abce6d4107764e18b0bccc5d2799434c" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37949 } } peers { permanent_uuid: "75040f489b5f42d3b5a1e3f264acdd2b" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 35509 } } }
I20260928 21:54:07.984788 5010 tablet_replica.cc:442] T 3a7d6d6b71494ecc9d1e5da480be64a5 P 26ef4abb4a48496e890d727f5e551105: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:54:08.004344 5012 raft_consensus.cc:482] T 3a7d6d6b71494ecc9d1e5da480be64a5 P abce6d4107764e18b0bccc5d2799434c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:54:08.004433 5012 raft_consensus.cc:504] T 3a7d6d6b71494ecc9d1e5da480be64a5 P abce6d4107764e18b0bccc5d2799434c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26ef4abb4a48496e890d727f5e551105" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 43373 } } peers { permanent_uuid: "abce6d4107764e18b0bccc5d2799434c" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37949 } } peers { permanent_uuid: "75040f489b5f42d3b5a1e3f264acdd2b" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 35509 } }
I20260928 21:54:08.004562 5012 leader_election.cc:288] T 3a7d6d6b71494ecc9d1e5da480be64a5 P abce6d4107764e18b0bccc5d2799434c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26ef4abb4a48496e890d727f5e551105 (127.2.182.65:43373), 75040f489b5f42d3b5a1e3f264acdd2b (127.2.182.67:35509)
I20260928 21:54:08.004767 4935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7d6d6b71494ecc9d1e5da480be64a5" candidate_uuid: "abce6d4107764e18b0bccc5d2799434c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75040f489b5f42d3b5a1e3f264acdd2b" is_pre_election: true
I20260928 21:54:08.004884 4935 raft_consensus.cc:2454] T 3a7d6d6b71494ecc9d1e5da480be64a5 P 75040f489b5f42d3b5a1e3f264acdd2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abce6d4107764e18b0bccc5d2799434c in term 0.
I20260928 21:54:08.004853 4655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7d6d6b71494ecc9d1e5da480be64a5" candidate_uuid: "abce6d4107764e18b0bccc5d2799434c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26ef4abb4a48496e890d727f5e551105" is_pre_election: true
I20260928 21:54:08.004926 4655 raft_consensus.cc:2454] T 3a7d6d6b71494ecc9d1e5da480be64a5 P 26ef4abb4a48496e890d727f5e551105 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abce6d4107764e18b0bccc5d2799434c in term 0.
I20260928 21:54:08.005080 4728 leader_election.cc:302] T 3a7d6d6b71494ecc9d1e5da480be64a5 P abce6d4107764e18b0bccc5d2799434c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26ef4abb4a48496e890d727f5e551105, abce6d4107764e18b0bccc5d2799434c; no voters:
I20260928 21:54:08.005189 5012 raft_consensus.cc:2781] T 3a7d6d6b71494ecc9d1e5da480be64a5 P abce6d4107764e18b0bccc5d2799434c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:54:08.005228 5012 raft_consensus.cc:482] T 3a7d6d6b71494ecc9d1e5da480be64a5 P abce6d4107764e18b0bccc5d2799434c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:54:08.005242 5012 raft_consensus.cc:3037] T 3a7d6d6b71494ecc9d1e5da480be64a5 P abce6d4107764e18b0bccc5d2799434c [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:54:08.005861 5012 raft_consensus.cc:504] T 3a7d6d6b71494ecc9d1e5da480be64a5 P abce6d4107764e18b0bccc5d2799434c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26ef4abb4a48496e890d727f5e551105" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 43373 } } peers { permanent_uuid: "abce6d4107764e18b0bccc5d2799434c" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37949 } } peers { permanent_uuid: "75040f489b5f42d3b5a1e3f264acdd2b" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 35509 } }
I20260928 21:54:08.006124 5012 leader_election.cc:288] T 3a7d6d6b71494ecc9d1e5da480be64a5 P abce6d4107764e18b0bccc5d2799434c [CANDIDATE]: Term 1 election: Requested vote from peers 26ef4abb4a48496e890d727f5e551105 (127.2.182.65:43373), 75040f489b5f42d3b5a1e3f264acdd2b (127.2.182.67:35509)
I20260928 21:54:08.006223 4655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7d6d6b71494ecc9d1e5da480be64a5" candidate_uuid: "abce6d4107764e18b0bccc5d2799434c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26ef4abb4a48496e890d727f5e551105"
I20260928 21:54:08.006306 4655 raft_consensus.cc:3037] T 3a7d6d6b71494ecc9d1e5da480be64a5 P 26ef4abb4a48496e890d727f5e551105 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:54:08.006294 4935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7d6d6b71494ecc9d1e5da480be64a5" candidate_uuid: "abce6d4107764e18b0bccc5d2799434c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75040f489b5f42d3b5a1e3f264acdd2b"
I20260928 21:54:08.006372 4935 raft_consensus.cc:3037] T 3a7d6d6b71494ecc9d1e5da480be64a5 P 75040f489b5f42d3b5a1e3f264acdd2b [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:54:08.006920 4655 raft_consensus.cc:2454] T 3a7d6d6b71494ecc9d1e5da480be64a5 P 26ef4abb4a48496e890d727f5e551105 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abce6d4107764e18b0bccc5d2799434c in term 1.
I20260928 21:54:08.007050 4935 raft_consensus.cc:2454] T 3a7d6d6b71494ecc9d1e5da480be64a5 P 75040f489b5f42d3b5a1e3f264acdd2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abce6d4107764e18b0bccc5d2799434c in term 1.
I20260928 21:54:08.007103 4728 leader_election.cc:302] T 3a7d6d6b71494ecc9d1e5da480be64a5 P abce6d4107764e18b0bccc5d2799434c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26ef4abb4a48496e890d727f5e551105, abce6d4107764e18b0bccc5d2799434c; no voters:
I20260928 21:54:08.007225 5012 raft_consensus.cc:2781] T 3a7d6d6b71494ecc9d1e5da480be64a5 P abce6d4107764e18b0bccc5d2799434c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:54:08.007282 5012 raft_consensus.cc:686] T 3a7d6d6b71494ecc9d1e5da480be64a5 P abce6d4107764e18b0bccc5d2799434c [term 1 LEADER]: Becoming Leader. State: Replica: abce6d4107764e18b0bccc5d2799434c, State: Running, Role: LEADER
I20260928 21:54:08.007350 5012 consensus_queue.cc:237] T 3a7d6d6b71494ecc9d1e5da480be64a5 P abce6d4107764e18b0bccc5d2799434c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26ef4abb4a48496e890d727f5e551105" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 43373 } } peers { permanent_uuid: "abce6d4107764e18b0bccc5d2799434c" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37949 } } peers { permanent_uuid: "75040f489b5f42d3b5a1e3f264acdd2b" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 35509 } }
I20260928 21:54:08.007786 5014 tablet_replica.cc:439] T 3a7d6d6b71494ecc9d1e5da480be64a5 P abce6d4107764e18b0bccc5d2799434c: TxnStatusTablet state changed. Reason: New leader abce6d4107764e18b0bccc5d2799434c. Latest consensus state: current_term: 1 leader_uuid: "abce6d4107764e18b0bccc5d2799434c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26ef4abb4a48496e890d727f5e551105" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 43373 } } peers { permanent_uuid: "abce6d4107764e18b0bccc5d2799434c" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37949 } } peers { permanent_uuid: "75040f489b5f42d3b5a1e3f264acdd2b" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 35509 } } }
I20260928 21:54:08.007899 5014 tablet_replica.cc:442] T 3a7d6d6b71494ecc9d1e5da480be64a5 P abce6d4107764e18b0bccc5d2799434c: This TxnStatusTablet replica's current role is: LEADER
I20260928 21:54:08.008168 5026 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 21:54:08.008174 4498 catalog_manager.cc:5495] T 3a7d6d6b71494ecc9d1e5da480be64a5 P abce6d4107764e18b0bccc5d2799434c reported cstate change: term changed from 0 to 1, leader changed from <none> to abce6d4107764e18b0bccc5d2799434c (127.2.182.66). New cstate: current_term: 1 leader_uuid: "abce6d4107764e18b0bccc5d2799434c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26ef4abb4a48496e890d727f5e551105" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 43373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "abce6d4107764e18b0bccc5d2799434c" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75040f489b5f42d3b5a1e3f264acdd2b" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 35509 } health_report { overall_health: UNKNOWN } } }
I20260928 21:54:08.008301 5026 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 21:54:08.015460 5003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:08.012530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44470 (local address 127.2.182.66:37949)
0928 21:54:08.012592 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:08.012594 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:08.012627 (+ 33us) server_negotiation.cc:407] Connection header received
0928 21:54:08.012659 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:08.012660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:08.012679 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:08.012743 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:54:08.012852 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:08.013638 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:08.014403 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:08.014699 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:08.014771 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:08.015027 (+ 256us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:54:08.015139 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:54:08.015222 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:08.015325 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 21:54:08.015381 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 21:54:08.015501 5021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:08.012417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.66:37949 (local address 127.0.0.1:44470)
0928 21:54:08.012510 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:08.012524 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:08.012583 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:08.012748 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:08.012749 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:08.012750 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 21:54:08.012795 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:08.012797 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:08.013728 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:08.013730 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:08.014322 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:08.014325 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:08.014793 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:08.014794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:08.014878 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:08.014905 (+ 27us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 21:54:08.015227 (+ 322us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 21:54:08.015238 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 21:54:08.015320 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 21:54:08.015412 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 21:54:08.021090 4655 raft_consensus.cc:1261] T 3a7d6d6b71494ecc9d1e5da480be64a5 P 26ef4abb4a48496e890d727f5e551105 [term 1 FOLLOWER]: Refusing update from remote peer abce6d4107764e18b0bccc5d2799434c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:54:08.021593 4935 raft_consensus.cc:1261] T 3a7d6d6b71494ecc9d1e5da480be64a5 P 75040f489b5f42d3b5a1e3f264acdd2b [term 1 FOLLOWER]: Refusing update from remote peer abce6d4107764e18b0bccc5d2799434c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:54:08.022135 5014 consensus_queue.cc:1035] T 3a7d6d6b71494ecc9d1e5da480be64a5 P abce6d4107764e18b0bccc5d2799434c [LEADER]: Connected to new peer: Peer: permanent_uuid: "26ef4abb4a48496e890d727f5e551105" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 43373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:54:08.022331 5012 consensus_queue.cc:1035] T 3a7d6d6b71494ecc9d1e5da480be64a5 P abce6d4107764e18b0bccc5d2799434c [LEADER]: Connected to new peer: Peer: permanent_uuid: "75040f489b5f42d3b5a1e3f264acdd2b" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:54:08.023641 5011 tablet_replica.cc:439] T 3a7d6d6b71494ecc9d1e5da480be64a5 P 75040f489b5f42d3b5a1e3f264acdd2b: TxnStatusTablet state changed. Reason: New leader abce6d4107764e18b0bccc5d2799434c. Latest consensus state: current_term: 1 leader_uuid: "abce6d4107764e18b0bccc5d2799434c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26ef4abb4a48496e890d727f5e551105" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 43373 } } peers { permanent_uuid: "abce6d4107764e18b0bccc5d2799434c" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37949 } } peers { permanent_uuid: "75040f489b5f42d3b5a1e3f264acdd2b" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 35509 } } }
I20260928 21:54:08.023749 5011 tablet_replica.cc:442] T 3a7d6d6b71494ecc9d1e5da480be64a5 P 75040f489b5f42d3b5a1e3f264acdd2b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:54:08.023797 5010 tablet_replica.cc:439] T 3a7d6d6b71494ecc9d1e5da480be64a5 P 26ef4abb4a48496e890d727f5e551105: TxnStatusTablet state changed. Reason: New leader abce6d4107764e18b0bccc5d2799434c. Latest consensus state: current_term: 1 leader_uuid: "abce6d4107764e18b0bccc5d2799434c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26ef4abb4a48496e890d727f5e551105" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 43373 } } peers { permanent_uuid: "abce6d4107764e18b0bccc5d2799434c" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37949 } } peers { permanent_uuid: "75040f489b5f42d3b5a1e3f264acdd2b" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 35509 } } }
I20260928 21:54:08.023880 5010 tablet_replica.cc:442] T 3a7d6d6b71494ecc9d1e5da480be64a5 P 26ef4abb4a48496e890d727f5e551105: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:54:08.024825 5012 tablet_replica.cc:439] T 3a7d6d6b71494ecc9d1e5da480be64a5 P abce6d4107764e18b0bccc5d2799434c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "abce6d4107764e18b0bccc5d2799434c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26ef4abb4a48496e890d727f5e551105" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 43373 } } peers { permanent_uuid: "abce6d4107764e18b0bccc5d2799434c" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37949 } } peers { permanent_uuid: "75040f489b5f42d3b5a1e3f264acdd2b" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 35509 } } }
I20260928 21:54:08.024914 5012 tablet_replica.cc:442] T 3a7d6d6b71494ecc9d1e5da480be64a5 P abce6d4107764e18b0bccc5d2799434c: This TxnStatusTablet replica's current role is: LEADER
I20260928 21:54:08.025178 5014 tablet_replica.cc:439] T 3a7d6d6b71494ecc9d1e5da480be64a5 P abce6d4107764e18b0bccc5d2799434c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "abce6d4107764e18b0bccc5d2799434c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26ef4abb4a48496e890d727f5e551105" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 43373 } } peers { permanent_uuid: "abce6d4107764e18b0bccc5d2799434c" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37949 } } peers { permanent_uuid: "75040f489b5f42d3b5a1e3f264acdd2b" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 35509 } } }
I20260928 21:54:08.025242 5014 tablet_replica.cc:442] T 3a7d6d6b71494ecc9d1e5da480be64a5 P abce6d4107764e18b0bccc5d2799434c: This TxnStatusTablet replica's current role is: LEADER
I20260928 21:54:08.025593 5011 tablet_replica.cc:439] T 3a7d6d6b71494ecc9d1e5da480be64a5 P 75040f489b5f42d3b5a1e3f264acdd2b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "abce6d4107764e18b0bccc5d2799434c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26ef4abb4a48496e890d727f5e551105" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 43373 } } peers { permanent_uuid: "abce6d4107764e18b0bccc5d2799434c" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37949 } } peers { permanent_uuid: "75040f489b5f42d3b5a1e3f264acdd2b" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 35509 } } }
I20260928 21:54:08.025635 5010 tablet_replica.cc:439] T 3a7d6d6b71494ecc9d1e5da480be64a5 P 26ef4abb4a48496e890d727f5e551105: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "abce6d4107764e18b0bccc5d2799434c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26ef4abb4a48496e890d727f5e551105" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 43373 } } peers { permanent_uuid: "abce6d4107764e18b0bccc5d2799434c" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37949 } } peers { permanent_uuid: "75040f489b5f42d3b5a1e3f264acdd2b" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 35509 } } }
I20260928 21:54:08.025689 5011 tablet_replica.cc:442] T 3a7d6d6b71494ecc9d1e5da480be64a5 P 75040f489b5f42d3b5a1e3f264acdd2b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:54:08.025694 5010 tablet_replica.cc:442] T 3a7d6d6b71494ecc9d1e5da480be64a5 P 26ef4abb4a48496e890d727f5e551105: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:54:08.025935 5031 mvcc.cc:205] Tried to move back new op lower bound from 7334430507091111936 to 7334430507038310400. Current Snapshot: MvccSnapshot[applied={T|T < 7334430507091111936}]
I20260928 21:54:08.026580 5030 mvcc.cc:205] Tried to move back new op lower bound from 7334430507091111936 to 7334430507038310400. Current Snapshot: MvccSnapshot[applied={T|T < 7334430507091111936 or (T in {7334430507091111936})}]
I20260928 21:54:08.030217 5003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:08.027692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44480 (local address 127.2.182.66:37949)
0928 21:54:08.027735 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:08.027737 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:08.027747 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:54:08.027796 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:08.027797 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:08.027806 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:08.027863 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:54:08.027972 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:08.028703 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:08.029505 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:08.029805 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:08.029872 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:08.029999 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:54:08.030032 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:54:08.030077 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:08.030122 (+ 45us) server_negotiation.cc:299] Negotiation successful
0928 21:54:08.030142 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 21:54:08.035598 5003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:08.032960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.66:48235 (local address 127.2.182.66:37949)
0928 21:54:08.032997 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:08.032999 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:08.033092 (+ 93us) server_negotiation.cc:407] Connection header received
0928 21:54:08.033154 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:08.033154 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:08.033162 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:08.033212 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:54:08.033352 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:08.033825 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:08.034905 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:08.035077 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:08.035134 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:08.035321 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:54:08.035386 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:54:08.035435 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:08.035495 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 21:54:08.035513 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 21:54:08.035666 4847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:08.032910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.66:37949 (local address 127.2.182.66:48235)
0928 21:54:08.033042 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:08.033050 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:08.033093 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:08.033228 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:08.033229 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:08.033230 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 21:54:08.033288 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:08.033290 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:08.033886 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:08.033887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:08.034829 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:08.034831 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:08.035149 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:08.035150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:08.035215 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:08.035232 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 21:54:08.035442 (+ 210us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 21:54:08.035444 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 21:54:08.035492 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 21:54:08.035534 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107}
I20260928 21:54:08.037868 4655 raft_consensus.cc:1261] T 5548267a9c6446fab81daad93c3a00fb P 26ef4abb4a48496e890d727f5e551105 [term 1 FOLLOWER]: Refusing update from remote peer abce6d4107764e18b0bccc5d2799434c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:54:08.037868 4935 raft_consensus.cc:1261] T 5548267a9c6446fab81daad93c3a00fb P 75040f489b5f42d3b5a1e3f264acdd2b [term 1 FOLLOWER]: Refusing update from remote peer abce6d4107764e18b0bccc5d2799434c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:54:08.038053 5035 consensus_queue.cc:1035] T 5548267a9c6446fab81daad93c3a00fb P abce6d4107764e18b0bccc5d2799434c [LEADER]: Connected to new peer: Peer: permanent_uuid: "75040f489b5f42d3b5a1e3f264acdd2b" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:54:08.038166 5014 consensus_queue.cc:1035] T 5548267a9c6446fab81daad93c3a00fb P abce6d4107764e18b0bccc5d2799434c [LEADER]: Connected to new peer: Peer: permanent_uuid: "26ef4abb4a48496e890d727f5e551105" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 43373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:54:08.039582 5029 mvcc.cc:205] Tried to move back new op lower bound from 7334430507161714688 to 7334430506760458240. Current Snapshot: MvccSnapshot[applied={T|T < 7334430507161714688}]
I20260928 21:54:08.049328 4497 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42846:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:54:08.049434 4497 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42846:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:54:08.050141 4497 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 39ad015d374848d084369d0f182d3f51: Sending DeleteTablet for 3 replicas of tablet 5548267a9c6446fab81daad93c3a00fb
I20260928 21:54:08.050414 4915 tablet_service.cc:1514] Processing DeleteTablet for tablet 5548267a9c6446fab81daad93c3a00fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:54:08 UTC) from {username='kudu', principal='kudu/127.2.182.126@KRBTEST.COM'} at 127.0.0.1:38864
I20260928 21:54:08.050449 4628 tablet_service.cc:1514] Processing DeleteTablet for tablet 5548267a9c6446fab81daad93c3a00fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:54:08 UTC) from {username='kudu', principal='kudu/127.2.182.126@KRBTEST.COM'} at 127.0.0.1:33448
I20260928 21:54:08.050479 4775 tablet_service.cc:1514] Processing DeleteTablet for tablet 5548267a9c6446fab81daad93c3a00fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:54:08 UTC) from {username='kudu', principal='kudu/127.2.182.126@KRBTEST.COM'} at 127.0.0.1:44464
I20260928 21:54:08.050664 5045 tablet_replica.cc:330] T 5548267a9c6446fab81daad93c3a00fb P abce6d4107764e18b0bccc5d2799434c: stopping tablet replica
I20260928 21:54:08.050770 5045 raft_consensus.cc:2229] T 5548267a9c6446fab81daad93c3a00fb P abce6d4107764e18b0bccc5d2799434c [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:54:08.050891 5045 raft_consensus.cc:2258] T 5548267a9c6446fab81daad93c3a00fb P abce6d4107764e18b0bccc5d2799434c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:54:08.051011 4497 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42846:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:54:08.051165 4497 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:54:08.051402 5046 tablet_replica.cc:330] T 5548267a9c6446fab81daad93c3a00fb P 26ef4abb4a48496e890d727f5e551105: stopping tablet replica
I20260928 21:54:08.051517 5046 raft_consensus.cc:2229] T 5548267a9c6446fab81daad93c3a00fb P 26ef4abb4a48496e890d727f5e551105 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:54:08.051578 5046 raft_consensus.cc:2258] T 5548267a9c6446fab81daad93c3a00fb P 26ef4abb4a48496e890d727f5e551105 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:54:08.051640 5044 tablet_replica.cc:330] T 5548267a9c6446fab81daad93c3a00fb P 75040f489b5f42d3b5a1e3f264acdd2b: stopping tablet replica
I20260928 21:54:08.051741 5044 raft_consensus.cc:2229] T 5548267a9c6446fab81daad93c3a00fb P 75040f489b5f42d3b5a1e3f264acdd2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:54:08.051789 5044 raft_consensus.cc:2258] T 5548267a9c6446fab81daad93c3a00fb P 75040f489b5f42d3b5a1e3f264acdd2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:54:08.052007 5045 ts_tablet_manager.cc:1905] T 5548267a9c6446fab81daad93c3a00fb P abce6d4107764e18b0bccc5d2799434c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:54:08.052179 5044 ts_tablet_manager.cc:1905] T 5548267a9c6446fab81daad93c3a00fb P 75040f489b5f42d3b5a1e3f264acdd2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:54:08.052877 4915 tablet_service.cc:1467] Processing CreateTablet for tablet 73c5ff3674044b4ebad77c94ca56dbe4 (DEFAULT_TABLE table=test-table [id=df044846521448f69db898fe02a09832]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:54:08.052980 4915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73c5ff3674044b4ebad77c94ca56dbe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:54:08.053954 5045 ts_tablet_manager.cc:1918] T 5548267a9c6446fab81daad93c3a00fb P abce6d4107764e18b0bccc5d2799434c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:54:08.053995 5044 ts_tablet_manager.cc:1918] T 5548267a9c6446fab81daad93c3a00fb P 75040f489b5f42d3b5a1e3f264acdd2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:54:08.054019 5045 log.cc:1196] T 5548267a9c6446fab81daad93c3a00fb P abce6d4107764e18b0bccc5d2799434c: Deleting WAL directory at /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/minicluster-data/ts-1/wal/wals/5548267a9c6446fab81daad93c3a00fb
I20260928 21:54:08.054061 5044 log.cc:1196] T 5548267a9c6446fab81daad93c3a00fb P 75040f489b5f42d3b5a1e3f264acdd2b: Deleting WAL directory at /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/minicluster-data/ts-2/wal/wals/5548267a9c6446fab81daad93c3a00fb
I20260928 21:54:08.054281 5004 tablet_bootstrap.cc:492] T 73c5ff3674044b4ebad77c94ca56dbe4 P 75040f489b5f42d3b5a1e3f264acdd2b: Bootstrap starting.
I20260928 21:54:08.054407 5044 ts_tablet_manager.cc:1939] T 5548267a9c6446fab81daad93c3a00fb P 75040f489b5f42d3b5a1e3f264acdd2b: Deleting consensus metadata
I20260928 21:54:08.054653 5004 tablet_bootstrap.cc:654] T 73c5ff3674044b4ebad77c94ca56dbe4 P 75040f489b5f42d3b5a1e3f264acdd2b: Neither blocks nor log segments found. Creating new log.
I20260928 21:54:08.054893 4775 tablet_service.cc:1467] Processing CreateTablet for tablet 73c5ff3674044b4ebad77c94ca56dbe4 (DEFAULT_TABLE table=test-table [id=df044846521448f69db898fe02a09832]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:54:08.055001 4775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73c5ff3674044b4ebad77c94ca56dbe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:54:08.055136 4628 tablet_service.cc:1467] Processing CreateTablet for tablet 73c5ff3674044b4ebad77c94ca56dbe4 (DEFAULT_TABLE table=test-table [id=df044846521448f69db898fe02a09832]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:54:08.055254 4628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73c5ff3674044b4ebad77c94ca56dbe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:54:08.055421 5045 ts_tablet_manager.cc:1939] T 5548267a9c6446fab81daad93c3a00fb P abce6d4107764e18b0bccc5d2799434c: Deleting consensus metadata
I20260928 21:54:08.056186 5046 ts_tablet_manager.cc:1905] T 5548267a9c6446fab81daad93c3a00fb P 26ef4abb4a48496e890d727f5e551105: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:54:08.056538 5007 tablet_bootstrap.cc:492] T 73c5ff3674044b4ebad77c94ca56dbe4 P abce6d4107764e18b0bccc5d2799434c: Bootstrap starting.
I20260928 21:54:08.056701 5005 tablet_bootstrap.cc:492] T 73c5ff3674044b4ebad77c94ca56dbe4 P 26ef4abb4a48496e890d727f5e551105: Bootstrap starting.
I20260928 21:54:08.056946 5007 tablet_bootstrap.cc:654] T 73c5ff3674044b4ebad77c94ca56dbe4 P abce6d4107764e18b0bccc5d2799434c: Neither blocks nor log segments found. Creating new log.
I20260928 21:54:08.057081 5005 tablet_bootstrap.cc:654] T 73c5ff3674044b4ebad77c94ca56dbe4 P 26ef4abb4a48496e890d727f5e551105: Neither blocks nor log segments found. Creating new log.
I20260928 21:54:08.057472 4484 catalog_manager.cc:4841] TS abce6d4107764e18b0bccc5d2799434c (127.2.182.66:37949): tablet 5548267a9c6446fab81daad93c3a00fb (table test-table [id=9e27fe05f25d49fd969310fd488d5172]) successfully deleted
I20260928 21:54:08.057866 4485 catalog_manager.cc:4841] TS 75040f489b5f42d3b5a1e3f264acdd2b (127.2.182.67:35509): tablet 5548267a9c6446fab81daad93c3a00fb (table test-table [id=9e27fe05f25d49fd969310fd488d5172]) successfully deleted
I20260928 21:54:08.058372 5046 ts_tablet_manager.cc:1918] T 5548267a9c6446fab81daad93c3a00fb P 26ef4abb4a48496e890d727f5e551105: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:54:08.058473 5046 log.cc:1196] T 5548267a9c6446fab81daad93c3a00fb P 26ef4abb4a48496e890d727f5e551105: Deleting WAL directory at /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/minicluster-data/ts-0/wal/wals/5548267a9c6446fab81daad93c3a00fb
I20260928 21:54:08.058476 5004 tablet_bootstrap.cc:492] T 73c5ff3674044b4ebad77c94ca56dbe4 P 75040f489b5f42d3b5a1e3f264acdd2b: No bootstrap required, opened a new log
I20260928 21:54:08.058521 5004 ts_tablet_manager.cc:1397] T 73c5ff3674044b4ebad77c94ca56dbe4 P 75040f489b5f42d3b5a1e3f264acdd2b: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260928 21:54:08.058662 5004 raft_consensus.cc:348] T 73c5ff3674044b4ebad77c94ca56dbe4 P 75040f489b5f42d3b5a1e3f264acdd2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26ef4abb4a48496e890d727f5e551105" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 43373 } } peers { permanent_uuid: "abce6d4107764e18b0bccc5d2799434c" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37949 } } peers { permanent_uuid: "75040f489b5f42d3b5a1e3f264acdd2b" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 35509 } }
I20260928 21:54:08.058727 5004 raft_consensus.cc:374] T 73c5ff3674044b4ebad77c94ca56dbe4 P 75040f489b5f42d3b5a1e3f264acdd2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:54:08.058741 5004 raft_consensus.cc:729] T 73c5ff3674044b4ebad77c94ca56dbe4 P 75040f489b5f42d3b5a1e3f264acdd2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75040f489b5f42d3b5a1e3f264acdd2b, State: Initialized, Role: FOLLOWER
I20260928 21:54:08.058787 5004 consensus_queue.cc:260] T 73c5ff3674044b4ebad77c94ca56dbe4 P 75040f489b5f42d3b5a1e3f264acdd2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26ef4abb4a48496e890d727f5e551105" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 43373 } } peers { permanent_uuid: "abce6d4107764e18b0bccc5d2799434c" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37949 } } peers { permanent_uuid: "75040f489b5f42d3b5a1e3f264acdd2b" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 35509 } }
I20260928 21:54:08.058843 5046 ts_tablet_manager.cc:1939] T 5548267a9c6446fab81daad93c3a00fb P 26ef4abb4a48496e890d727f5e551105: Deleting consensus metadata
I20260928 21:54:08.058880 5004 ts_tablet_manager.cc:1428] T 73c5ff3674044b4ebad77c94ca56dbe4 P 75040f489b5f42d3b5a1e3f264acdd2b: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 21:54:08.059255 4484 catalog_manager.cc:4841] TS 26ef4abb4a48496e890d727f5e551105 (127.2.182.65:43373): tablet 5548267a9c6446fab81daad93c3a00fb (table test-table [id=9e27fe05f25d49fd969310fd488d5172]) successfully deleted
I20260928 21:54:08.063745 5005 tablet_bootstrap.cc:492] T 73c5ff3674044b4ebad77c94ca56dbe4 P 26ef4abb4a48496e890d727f5e551105: No bootstrap required, opened a new log
I20260928 21:54:08.063746 5007 tablet_bootstrap.cc:492] T 73c5ff3674044b4ebad77c94ca56dbe4 P abce6d4107764e18b0bccc5d2799434c: No bootstrap required, opened a new log
I20260928 21:54:08.063804 5007 ts_tablet_manager.cc:1397] T 73c5ff3674044b4ebad77c94ca56dbe4 P abce6d4107764e18b0bccc5d2799434c: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260928 21:54:08.063805 5005 ts_tablet_manager.cc:1397] T 73c5ff3674044b4ebad77c94ca56dbe4 P 26ef4abb4a48496e890d727f5e551105: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260928 21:54:08.063953 5005 raft_consensus.cc:348] T 73c5ff3674044b4ebad77c94ca56dbe4 P 26ef4abb4a48496e890d727f5e551105 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26ef4abb4a48496e890d727f5e551105" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 43373 } } peers { permanent_uuid: "abce6d4107764e18b0bccc5d2799434c" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37949 } } peers { permanent_uuid: "75040f489b5f42d3b5a1e3f264acdd2b" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 35509 } }
I20260928 21:54:08.063941 5007 raft_consensus.cc:348] T 73c5ff3674044b4ebad77c94ca56dbe4 P abce6d4107764e18b0bccc5d2799434c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26ef4abb4a48496e890d727f5e551105" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 43373 } } peers { permanent_uuid: "abce6d4107764e18b0bccc5d2799434c" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37949 } } peers { permanent_uuid: "75040f489b5f42d3b5a1e3f264acdd2b" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 35509 } }
I20260928 21:54:08.064016 5005 raft_consensus.cc:374] T 73c5ff3674044b4ebad77c94ca56dbe4 P 26ef4abb4a48496e890d727f5e551105 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:54:08.064020 5007 raft_consensus.cc:374] T 73c5ff3674044b4ebad77c94ca56dbe4 P abce6d4107764e18b0bccc5d2799434c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:54:08.064034 5005 raft_consensus.cc:729] T 73c5ff3674044b4ebad77c94ca56dbe4 P 26ef4abb4a48496e890d727f5e551105 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26ef4abb4a48496e890d727f5e551105, State: Initialized, Role: FOLLOWER
I20260928 21:54:08.064040 5007 raft_consensus.cc:729] T 73c5ff3674044b4ebad77c94ca56dbe4 P abce6d4107764e18b0bccc5d2799434c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abce6d4107764e18b0bccc5d2799434c, State: Initialized, Role: FOLLOWER
I20260928 21:54:08.064085 5005 consensus_queue.cc:260] T 73c5ff3674044b4ebad77c94ca56dbe4 P 26ef4abb4a48496e890d727f5e551105 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26ef4abb4a48496e890d727f5e551105" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 43373 } } peers { permanent_uuid: "abce6d4107764e18b0bccc5d2799434c" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37949 } } peers { permanent_uuid: "75040f489b5f42d3b5a1e3f264acdd2b" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 35509 } }
I20260928 21:54:08.064090 5007 consensus_queue.cc:260] T 73c5ff3674044b4ebad77c94ca56dbe4 P abce6d4107764e18b0bccc5d2799434c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26ef4abb4a48496e890d727f5e551105" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 43373 } } peers { permanent_uuid: "abce6d4107764e18b0bccc5d2799434c" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37949 } } peers { permanent_uuid: "75040f489b5f42d3b5a1e3f264acdd2b" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 35509 } }
I20260928 21:54:08.064172 5007 ts_tablet_manager.cc:1428] T 73c5ff3674044b4ebad77c94ca56dbe4 P abce6d4107764e18b0bccc5d2799434c: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 21:54:08.064180 5005 ts_tablet_manager.cc:1428] T 73c5ff3674044b4ebad77c94ca56dbe4 P 26ef4abb4a48496e890d727f5e551105: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:08.176980 5035 raft_consensus.cc:482] T 73c5ff3674044b4ebad77c94ca56dbe4 P abce6d4107764e18b0bccc5d2799434c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:54:08.177096 5035 raft_consensus.cc:504] T 73c5ff3674044b4ebad77c94ca56dbe4 P abce6d4107764e18b0bccc5d2799434c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26ef4abb4a48496e890d727f5e551105" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 43373 } } peers { permanent_uuid: "abce6d4107764e18b0bccc5d2799434c" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37949 } } peers { permanent_uuid: "75040f489b5f42d3b5a1e3f264acdd2b" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 35509 } }
I20260928 21:54:08.177278 5035 leader_election.cc:288] T 73c5ff3674044b4ebad77c94ca56dbe4 P abce6d4107764e18b0bccc5d2799434c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26ef4abb4a48496e890d727f5e551105 (127.2.182.65:43373), 75040f489b5f42d3b5a1e3f264acdd2b (127.2.182.67:35509)
I20260928 21:54:08.177506 4935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73c5ff3674044b4ebad77c94ca56dbe4" candidate_uuid: "abce6d4107764e18b0bccc5d2799434c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75040f489b5f42d3b5a1e3f264acdd2b" is_pre_election: true
I20260928 21:54:08.177505 4655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73c5ff3674044b4ebad77c94ca56dbe4" candidate_uuid: "abce6d4107764e18b0bccc5d2799434c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26ef4abb4a48496e890d727f5e551105" is_pre_election: true
I20260928 21:54:08.177601 4935 raft_consensus.cc:2454] T 73c5ff3674044b4ebad77c94ca56dbe4 P 75040f489b5f42d3b5a1e3f264acdd2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abce6d4107764e18b0bccc5d2799434c in term 0.
I20260928 21:54:08.177611 4655 raft_consensus.cc:2454] T 73c5ff3674044b4ebad77c94ca56dbe4 P 26ef4abb4a48496e890d727f5e551105 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abce6d4107764e18b0bccc5d2799434c in term 0.
I20260928 21:54:08.177749 4729 leader_election.cc:302] T 73c5ff3674044b4ebad77c94ca56dbe4 P abce6d4107764e18b0bccc5d2799434c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75040f489b5f42d3b5a1e3f264acdd2b, abce6d4107764e18b0bccc5d2799434c; no voters:
I20260928 21:54:08.177835 5035 raft_consensus.cc:2781] T 73c5ff3674044b4ebad77c94ca56dbe4 P abce6d4107764e18b0bccc5d2799434c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:54:08.177870 5035 raft_consensus.cc:482] T 73c5ff3674044b4ebad77c94ca56dbe4 P abce6d4107764e18b0bccc5d2799434c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:54:08.177888 5035 raft_consensus.cc:3037] T 73c5ff3674044b4ebad77c94ca56dbe4 P abce6d4107764e18b0bccc5d2799434c [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:54:08.178740 5035 raft_consensus.cc:504] T 73c5ff3674044b4ebad77c94ca56dbe4 P abce6d4107764e18b0bccc5d2799434c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26ef4abb4a48496e890d727f5e551105" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 43373 } } peers { permanent_uuid: "abce6d4107764e18b0bccc5d2799434c" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37949 } } peers { permanent_uuid: "75040f489b5f42d3b5a1e3f264acdd2b" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 35509 } }
I20260928 21:54:08.178885 5035 leader_election.cc:288] T 73c5ff3674044b4ebad77c94ca56dbe4 P abce6d4107764e18b0bccc5d2799434c [CANDIDATE]: Term 1 election: Requested vote from peers 26ef4abb4a48496e890d727f5e551105 (127.2.182.65:43373), 75040f489b5f42d3b5a1e3f264acdd2b (127.2.182.67:35509)
I20260928 21:54:08.179082 4935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73c5ff3674044b4ebad77c94ca56dbe4" candidate_uuid: "abce6d4107764e18b0bccc5d2799434c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75040f489b5f42d3b5a1e3f264acdd2b"
I20260928 21:54:08.179082 4655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73c5ff3674044b4ebad77c94ca56dbe4" candidate_uuid: "abce6d4107764e18b0bccc5d2799434c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26ef4abb4a48496e890d727f5e551105"
I20260928 21:54:08.179167 4935 raft_consensus.cc:3037] T 73c5ff3674044b4ebad77c94ca56dbe4 P 75040f489b5f42d3b5a1e3f264acdd2b [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:54:08.179167 4655 raft_consensus.cc:3037] T 73c5ff3674044b4ebad77c94ca56dbe4 P 26ef4abb4a48496e890d727f5e551105 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:54:08.179903 4655 raft_consensus.cc:2454] T 73c5ff3674044b4ebad77c94ca56dbe4 P 26ef4abb4a48496e890d727f5e551105 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abce6d4107764e18b0bccc5d2799434c in term 1.
I20260928 21:54:08.179903 4935 raft_consensus.cc:2454] T 73c5ff3674044b4ebad77c94ca56dbe4 P 75040f489b5f42d3b5a1e3f264acdd2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abce6d4107764e18b0bccc5d2799434c in term 1.
I20260928 21:54:08.180084 4729 leader_election.cc:302] T 73c5ff3674044b4ebad77c94ca56dbe4 P abce6d4107764e18b0bccc5d2799434c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75040f489b5f42d3b5a1e3f264acdd2b, abce6d4107764e18b0bccc5d2799434c; no voters:
I20260928 21:54:08.180193 5035 raft_consensus.cc:2781] T 73c5ff3674044b4ebad77c94ca56dbe4 P abce6d4107764e18b0bccc5d2799434c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:54:08.180260 5035 raft_consensus.cc:686] T 73c5ff3674044b4ebad77c94ca56dbe4 P abce6d4107764e18b0bccc5d2799434c [term 1 LEADER]: Becoming Leader. State: Replica: abce6d4107764e18b0bccc5d2799434c, State: Running, Role: LEADER
I20260928 21:54:08.180307 5035 consensus_queue.cc:237] T 73c5ff3674044b4ebad77c94ca56dbe4 P abce6d4107764e18b0bccc5d2799434c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26ef4abb4a48496e890d727f5e551105" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 43373 } } peers { permanent_uuid: "abce6d4107764e18b0bccc5d2799434c" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37949 } } peers { permanent_uuid: "75040f489b5f42d3b5a1e3f264acdd2b" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 35509 } }
I20260928 21:54:08.181077 4496 catalog_manager.cc:5495] T 73c5ff3674044b4ebad77c94ca56dbe4 P abce6d4107764e18b0bccc5d2799434c reported cstate change: term changed from 0 to 1, leader changed from <none> to abce6d4107764e18b0bccc5d2799434c (127.2.182.66). New cstate: current_term: 1 leader_uuid: "abce6d4107764e18b0bccc5d2799434c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26ef4abb4a48496e890d727f5e551105" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 43373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "abce6d4107764e18b0bccc5d2799434c" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75040f489b5f42d3b5a1e3f264acdd2b" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 35509 } health_report { overall_health: UNKNOWN } } }
I20260928 21:54:08.205292 4935 raft_consensus.cc:1261] T 73c5ff3674044b4ebad77c94ca56dbe4 P 75040f489b5f42d3b5a1e3f264acdd2b [term 1 FOLLOWER]: Refusing update from remote peer abce6d4107764e18b0bccc5d2799434c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:54:08.205292 4655 raft_consensus.cc:1261] T 73c5ff3674044b4ebad77c94ca56dbe4 P 26ef4abb4a48496e890d727f5e551105 [term 1 FOLLOWER]: Refusing update from remote peer abce6d4107764e18b0bccc5d2799434c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:54:08.205515 5035 consensus_queue.cc:1035] T 73c5ff3674044b4ebad77c94ca56dbe4 P abce6d4107764e18b0bccc5d2799434c [LEADER]: Connected to new peer: Peer: permanent_uuid: "75040f489b5f42d3b5a1e3f264acdd2b" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:54:08.205650 5014 consensus_queue.cc:1035] T 73c5ff3674044b4ebad77c94ca56dbe4 P abce6d4107764e18b0bccc5d2799434c [LEADER]: Connected to new peer: Peer: permanent_uuid: "26ef4abb4a48496e890d727f5e551105" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 43373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 21:54:08.210440 5029 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 21:54:09.730494 4496 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42846:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:54:09.730593 4496 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42846:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:54:09.731514 4496 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 39ad015d374848d084369d0f182d3f51: Sending DeleteTablet for 3 replicas of tablet 73c5ff3674044b4ebad77c94ca56dbe4
I20260928 21:54:09.731748 4628 tablet_service.cc:1514] Processing DeleteTablet for tablet 73c5ff3674044b4ebad77c94ca56dbe4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:54:09 UTC) from {username='kudu', principal='kudu/127.2.182.126@KRBTEST.COM'} at 127.0.0.1:33448
I20260928 21:54:09.731760 4774 tablet_service.cc:1514] Processing DeleteTablet for tablet 73c5ff3674044b4ebad77c94ca56dbe4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:54:09 UTC) from {username='kudu', principal='kudu/127.2.182.126@KRBTEST.COM'} at 127.0.0.1:44464
I20260928 21:54:09.731822 4915 tablet_service.cc:1514] Processing DeleteTablet for tablet 73c5ff3674044b4ebad77c94ca56dbe4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:54:09 UTC) from {username='kudu', principal='kudu/127.2.182.126@KRBTEST.COM'} at 127.0.0.1:38864
I20260928 21:54:09.732077 5073 tablet_replica.cc:330] T 73c5ff3674044b4ebad77c94ca56dbe4 P 26ef4abb4a48496e890d727f5e551105: stopping tablet replica
I20260928 21:54:09.732121 5073 raft_consensus.cc:2229] T 73c5ff3674044b4ebad77c94ca56dbe4 P 26ef4abb4a48496e890d727f5e551105 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:54:09.732146 5073 raft_consensus.cc:2258] T 73c5ff3674044b4ebad77c94ca56dbe4 P 26ef4abb4a48496e890d727f5e551105 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:54:09.732384 5073 ts_tablet_manager.cc:1905] T 73c5ff3674044b4ebad77c94ca56dbe4 P 26ef4abb4a48496e890d727f5e551105: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:54:09.732401 5074 tablet_replica.cc:330] T 73c5ff3674044b4ebad77c94ca56dbe4 P 75040f489b5f42d3b5a1e3f264acdd2b: stopping tablet replica
I20260928 21:54:09.732467 5074 raft_consensus.cc:2229] T 73c5ff3674044b4ebad77c94ca56dbe4 P 75040f489b5f42d3b5a1e3f264acdd2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:54:09.732534 5074 raft_consensus.cc:2258] T 73c5ff3674044b4ebad77c94ca56dbe4 P 75040f489b5f42d3b5a1e3f264acdd2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:54:09.733867 5074 ts_tablet_manager.cc:1905] T 73c5ff3674044b4ebad77c94ca56dbe4 P 75040f489b5f42d3b5a1e3f264acdd2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:54:09.733915 5073 ts_tablet_manager.cc:1918] T 73c5ff3674044b4ebad77c94ca56dbe4 P 26ef4abb4a48496e890d727f5e551105: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 21:54:09.733958 5073 log.cc:1196] T 73c5ff3674044b4ebad77c94ca56dbe4 P 26ef4abb4a48496e890d727f5e551105: Deleting WAL directory at /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/minicluster-data/ts-0/wal/wals/73c5ff3674044b4ebad77c94ca56dbe4
I20260928 21:54:09.734156 5073 ts_tablet_manager.cc:1939] T 73c5ff3674044b4ebad77c94ca56dbe4 P 26ef4abb4a48496e890d727f5e551105: Deleting consensus metadata
I20260928 21:54:09.734458 4484 catalog_manager.cc:4841] TS 26ef4abb4a48496e890d727f5e551105 (127.2.182.65:43373): tablet 73c5ff3674044b4ebad77c94ca56dbe4 (table test-table [id=df044846521448f69db898fe02a09832]) successfully deleted
I20260928 21:54:09.735116 5075 tablet_replica.cc:330] T 73c5ff3674044b4ebad77c94ca56dbe4 P abce6d4107764e18b0bccc5d2799434c: stopping tablet replica
I20260928 21:54:09.735183 5075 raft_consensus.cc:2229] T 73c5ff3674044b4ebad77c94ca56dbe4 P abce6d4107764e18b0bccc5d2799434c [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:54:09.735251 5075 raft_consensus.cc:2258] T 73c5ff3674044b4ebad77c94ca56dbe4 P abce6d4107764e18b0bccc5d2799434c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:54:09.735441 5074 ts_tablet_manager.cc:1918] T 73c5ff3674044b4ebad77c94ca56dbe4 P 75040f489b5f42d3b5a1e3f264acdd2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 21:54:09.735498 5074 log.cc:1196] T 73c5ff3674044b4ebad77c94ca56dbe4 P 75040f489b5f42d3b5a1e3f264acdd2b: Deleting WAL directory at /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/minicluster-data/ts-2/wal/wals/73c5ff3674044b4ebad77c94ca56dbe4
I20260928 21:54:09.735663 5074 ts_tablet_manager.cc:1939] T 73c5ff3674044b4ebad77c94ca56dbe4 P 75040f489b5f42d3b5a1e3f264acdd2b: Deleting consensus metadata
I20260928 21:54:09.735962 4485 catalog_manager.cc:4841] TS 75040f489b5f42d3b5a1e3f264acdd2b (127.2.182.67:35509): tablet 73c5ff3674044b4ebad77c94ca56dbe4 (table test-table [id=df044846521448f69db898fe02a09832]) successfully deleted
I20260928 21:54:09.735973 5075 ts_tablet_manager.cc:1905] T 73c5ff3674044b4ebad77c94ca56dbe4 P abce6d4107764e18b0bccc5d2799434c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:54:09.737355 5075 ts_tablet_manager.cc:1918] T 73c5ff3674044b4ebad77c94ca56dbe4 P abce6d4107764e18b0bccc5d2799434c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 21:54:09.737418 5075 log.cc:1196] T 73c5ff3674044b4ebad77c94ca56dbe4 P abce6d4107764e18b0bccc5d2799434c: Deleting WAL directory at /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/minicluster-data/ts-1/wal/wals/73c5ff3674044b4ebad77c94ca56dbe4
I20260928 21:54:09.737629 5075 ts_tablet_manager.cc:1939] T 73c5ff3674044b4ebad77c94ca56dbe4 P abce6d4107764e18b0bccc5d2799434c: Deleting consensus metadata
I20260928 21:54:09.737937 4484 catalog_manager.cc:4841] TS abce6d4107764e18b0bccc5d2799434c (127.2.182.66:37949): tablet 73c5ff3674044b4ebad77c94ca56dbe4 (table test-table [id=df044846521448f69db898fe02a09832]) successfully deleted
Sep 28 21:54:09 dist-test-slave-k6lz krb5kdc[4453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632449, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 21:54:09 dist-test-slave-k6lz krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632449, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:54:09.754428 5083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:09.747096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42860 (local address 127.2.182.126:44601)
0928 21:54:09.747394 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:09.747396 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:09.747416 (+ 20us) server_negotiation.cc:407] Connection header received
0928 21:54:09.747446 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:09.747447 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:09.747459 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:09.747557 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:09.748087 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:09.748612 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:09.749396 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:09.749590 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:09.749635 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:09.751995 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:09.751997 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:09.751997 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:09.752005 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:09.753371 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:09.753708 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:09.753709 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:09.753709 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:09.753764 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:09.753922 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:09.753923 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:09.753923 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:09.754031 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:09.754120 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:09.754239 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 21:54:09.754323 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":85,"threads_started":1}
I20260928 21:54:09.756168 4496 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42860:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:54:09.756345 4496 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:54:09.757756 4628 tablet_service.cc:1467] Processing CreateTablet for tablet 22f6090263a14cde98886475acecba83 (DEFAULT_TABLE table=test-table [id=7d5b01c9270647c9b5590d40a033f479]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:54:09.757803 4915 tablet_service.cc:1467] Processing CreateTablet for tablet 22f6090263a14cde98886475acecba83 (DEFAULT_TABLE table=test-table [id=7d5b01c9270647c9b5590d40a033f479]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:54:09.757822 4774 tablet_service.cc:1467] Processing CreateTablet for tablet 22f6090263a14cde98886475acecba83 (DEFAULT_TABLE table=test-table [id=7d5b01c9270647c9b5590d40a033f479]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:54:09.757897 4915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22f6090263a14cde98886475acecba83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:54:09.757901 4628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22f6090263a14cde98886475acecba83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:54:09.757970 4774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22f6090263a14cde98886475acecba83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:54:09.759256 5086 tablet_bootstrap.cc:492] T 22f6090263a14cde98886475acecba83 P abce6d4107764e18b0bccc5d2799434c: Bootstrap starting.
I20260928 21:54:09.759279 5084 tablet_bootstrap.cc:492] T 22f6090263a14cde98886475acecba83 P 26ef4abb4a48496e890d727f5e551105: Bootstrap starting.
I20260928 21:54:09.759624 5084 tablet_bootstrap.cc:654] T 22f6090263a14cde98886475acecba83 P 26ef4abb4a48496e890d727f5e551105: Neither blocks nor log segments found. Creating new log.
I20260928 21:54:09.759648 5086 tablet_bootstrap.cc:654] T 22f6090263a14cde98886475acecba83 P abce6d4107764e18b0bccc5d2799434c: Neither blocks nor log segments found. Creating new log.
I20260928 21:54:09.759917 5085 tablet_bootstrap.cc:492] T 22f6090263a14cde98886475acecba83 P 75040f489b5f42d3b5a1e3f264acdd2b: Bootstrap starting.
I20260928 21:54:09.760164 5084 tablet_bootstrap.cc:492] T 22f6090263a14cde98886475acecba83 P 26ef4abb4a48496e890d727f5e551105: No bootstrap required, opened a new log
I20260928 21:54:09.760164 5086 tablet_bootstrap.cc:492] T 22f6090263a14cde98886475acecba83 P abce6d4107764e18b0bccc5d2799434c: No bootstrap required, opened a new log
I20260928 21:54:09.760216 5086 ts_tablet_manager.cc:1397] T 22f6090263a14cde98886475acecba83 P abce6d4107764e18b0bccc5d2799434c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:54:09.760216 5084 ts_tablet_manager.cc:1397] T 22f6090263a14cde98886475acecba83 P 26ef4abb4a48496e890d727f5e551105: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:54:09.760283 5085 tablet_bootstrap.cc:654] T 22f6090263a14cde98886475acecba83 P 75040f489b5f42d3b5a1e3f264acdd2b: Neither blocks nor log segments found. Creating new log.
I20260928 21:54:09.760320 5086 raft_consensus.cc:348] T 22f6090263a14cde98886475acecba83 P abce6d4107764e18b0bccc5d2799434c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26ef4abb4a48496e890d727f5e551105" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 43373 } } peers { permanent_uuid: "abce6d4107764e18b0bccc5d2799434c" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37949 } } peers { permanent_uuid: "75040f489b5f42d3b5a1e3f264acdd2b" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 35509 } }
I20260928 21:54:09.760353 5084 raft_consensus.cc:348] T 22f6090263a14cde98886475acecba83 P 26ef4abb4a48496e890d727f5e551105 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26ef4abb4a48496e890d727f5e551105" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 43373 } } peers { permanent_uuid: "abce6d4107764e18b0bccc5d2799434c" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37949 } } peers { permanent_uuid: "75040f489b5f42d3b5a1e3f264acdd2b" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 35509 } }
I20260928 21:54:09.760396 5086 raft_consensus.cc:374] T 22f6090263a14cde98886475acecba83 P abce6d4107764e18b0bccc5d2799434c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:54:09.760408 5084 raft_consensus.cc:374] T 22f6090263a14cde98886475acecba83 P 26ef4abb4a48496e890d727f5e551105 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:54:09.760416 5086 raft_consensus.cc:729] T 22f6090263a14cde98886475acecba83 P abce6d4107764e18b0bccc5d2799434c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abce6d4107764e18b0bccc5d2799434c, State: Initialized, Role: FOLLOWER
I20260928 21:54:09.760434 5084 raft_consensus.cc:729] T 22f6090263a14cde98886475acecba83 P 26ef4abb4a48496e890d727f5e551105 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26ef4abb4a48496e890d727f5e551105, State: Initialized, Role: FOLLOWER
I20260928 21:54:09.760481 5086 consensus_queue.cc:260] T 22f6090263a14cde98886475acecba83 P abce6d4107764e18b0bccc5d2799434c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26ef4abb4a48496e890d727f5e551105" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 43373 } } peers { permanent_uuid: "abce6d4107764e18b0bccc5d2799434c" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37949 } } peers { permanent_uuid: "75040f489b5f42d3b5a1e3f264acdd2b" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 35509 } }
I20260928 21:54:09.760489 5084 consensus_queue.cc:260] T 22f6090263a14cde98886475acecba83 P 26ef4abb4a48496e890d727f5e551105 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26ef4abb4a48496e890d727f5e551105" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 43373 } } peers { permanent_uuid: "abce6d4107764e18b0bccc5d2799434c" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37949 } } peers { permanent_uuid: "75040f489b5f42d3b5a1e3f264acdd2b" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 35509 } }
I20260928 21:54:09.760586 5086 ts_tablet_manager.cc:1428] T 22f6090263a14cde98886475acecba83 P abce6d4107764e18b0bccc5d2799434c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:09.760653 5084 ts_tablet_manager.cc:1428] T 22f6090263a14cde98886475acecba83 P 26ef4abb4a48496e890d727f5e551105: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 21:54:09.760856 5085 tablet_bootstrap.cc:492] T 22f6090263a14cde98886475acecba83 P 75040f489b5f42d3b5a1e3f264acdd2b: No bootstrap required, opened a new log
I20260928 21:54:09.760895 5085 ts_tablet_manager.cc:1397] T 22f6090263a14cde98886475acecba83 P 75040f489b5f42d3b5a1e3f264acdd2b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:54:09.761005 5085 raft_consensus.cc:348] T 22f6090263a14cde98886475acecba83 P 75040f489b5f42d3b5a1e3f264acdd2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26ef4abb4a48496e890d727f5e551105" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 43373 } } peers { permanent_uuid: "abce6d4107764e18b0bccc5d2799434c" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37949 } } peers { permanent_uuid: "75040f489b5f42d3b5a1e3f264acdd2b" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 35509 } }
I20260928 21:54:09.761063 5085 raft_consensus.cc:374] T 22f6090263a14cde98886475acecba83 P 75040f489b5f42d3b5a1e3f264acdd2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:54:09.761081 5085 raft_consensus.cc:729] T 22f6090263a14cde98886475acecba83 P 75040f489b5f42d3b5a1e3f264acdd2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75040f489b5f42d3b5a1e3f264acdd2b, State: Initialized, Role: FOLLOWER
I20260928 21:54:09.761117 5085 consensus_queue.cc:260] T 22f6090263a14cde98886475acecba83 P 75040f489b5f42d3b5a1e3f264acdd2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26ef4abb4a48496e890d727f5e551105" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 43373 } } peers { permanent_uuid: "abce6d4107764e18b0bccc5d2799434c" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37949 } } peers { permanent_uuid: "75040f489b5f42d3b5a1e3f264acdd2b" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 35509 } }
I20260928 21:54:09.761266 5085 ts_tablet_manager.cc:1428] T 22f6090263a14cde98886475acecba83 P 75040f489b5f42d3b5a1e3f264acdd2b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:09.856159 5090 raft_consensus.cc:482] T 22f6090263a14cde98886475acecba83 P 26ef4abb4a48496e890d727f5e551105 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:54:09.856266 5090 raft_consensus.cc:504] T 22f6090263a14cde98886475acecba83 P 26ef4abb4a48496e890d727f5e551105 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26ef4abb4a48496e890d727f5e551105" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 43373 } } peers { permanent_uuid: "abce6d4107764e18b0bccc5d2799434c" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37949 } } peers { permanent_uuid: "75040f489b5f42d3b5a1e3f264acdd2b" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 35509 } }
I20260928 21:54:09.856587 5090 leader_election.cc:288] T 22f6090263a14cde98886475acecba83 P 26ef4abb4a48496e890d727f5e551105 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers abce6d4107764e18b0bccc5d2799434c (127.2.182.66:37949), 75040f489b5f42d3b5a1e3f264acdd2b (127.2.182.67:35509)
I20260928 21:54:09.859650 5095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:09.856897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.65:35905 (local address 127.2.182.67:35509)
0928 21:54:09.857036 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:09.857038 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:09.857129 (+ 91us) server_negotiation.cc:407] Connection header received
0928 21:54:09.857205 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:09.857206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:09.857216 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:09.857277 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:54:09.857433 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:09.857974 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:09.858959 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:09.859321 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:09.859411 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:09.859420 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:09.859536 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 21:54:09.859576 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":85,"threads_started":1}
I20260928 21:54:09.859661 5093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:09.856837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.67:35509 (local address 127.2.182.65:35905)
0928 21:54:09.857048 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:09.857057 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:09.857131 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:09.857291 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:09.857292 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:09.857294 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:54:09.857371 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:09.857372 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:09.858066 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:09.858067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:09.858866 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:09.858868 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:09.859407 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:09.859408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:09.859477 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:09.859482 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 21:54:09.859526 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 21:54:09.859567 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":72,"threads_started":1}
I20260928 21:54:09.859881 5094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:09.856840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.65:59385 (local address 127.2.182.66:37949)
0928 21:54:09.857011 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:09.857012 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:09.857037 (+ 25us) server_negotiation.cc:407] Connection header received
0928 21:54:09.857103 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:09.857104 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:09.857114 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:09.857172 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:54:09.857327 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:09.857804 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:09.859082 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:09.859475 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:09.859538 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:09.859547 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:09.859771 (+ 224us) server_negotiation.cc:299] Negotiation successful
0928 21:54:09.859816 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":118,"threads_started":1}
I20260928 21:54:09.859809 5092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:09.856784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.66:37949 (local address 127.2.182.65:59385)
0928 21:54:09.856967 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:09.856979 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:09.857038 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:09.857190 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:09.857191 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:09.857193 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:54:09.857263 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:09.857264 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:09.857882 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:09.857883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:09.858938 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:09.858941 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:09.859548 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:09.859549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:09.859616 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:09.859619 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:54:09.859663 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 21:54:09.859701 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":115,"threads_started":1}
I20260928 21:54:09.860232 4935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22f6090263a14cde98886475acecba83" candidate_uuid: "26ef4abb4a48496e890d727f5e551105" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75040f489b5f42d3b5a1e3f264acdd2b" is_pre_election: true
I20260928 21:54:09.860246 4795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22f6090263a14cde98886475acecba83" candidate_uuid: "26ef4abb4a48496e890d727f5e551105" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abce6d4107764e18b0bccc5d2799434c" is_pre_election: true
I20260928 21:54:09.860328 4935 raft_consensus.cc:2454] T 22f6090263a14cde98886475acecba83 P 75040f489b5f42d3b5a1e3f264acdd2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26ef4abb4a48496e890d727f5e551105 in term 0.
I20260928 21:54:09.860355 4795 raft_consensus.cc:2454] T 22f6090263a14cde98886475acecba83 P abce6d4107764e18b0bccc5d2799434c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26ef4abb4a48496e890d727f5e551105 in term 0.
I20260928 21:54:09.860515 4588 leader_election.cc:302] T 22f6090263a14cde98886475acecba83 P 26ef4abb4a48496e890d727f5e551105 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26ef4abb4a48496e890d727f5e551105, abce6d4107764e18b0bccc5d2799434c; no voters:
I20260928 21:54:09.860646 5090 raft_consensus.cc:2781] T 22f6090263a14cde98886475acecba83 P 26ef4abb4a48496e890d727f5e551105 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:54:09.860678 5090 raft_consensus.cc:482] T 22f6090263a14cde98886475acecba83 P 26ef4abb4a48496e890d727f5e551105 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:54:09.860705 5090 raft_consensus.cc:3037] T 22f6090263a14cde98886475acecba83 P 26ef4abb4a48496e890d727f5e551105 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:54:09.861474 5090 raft_consensus.cc:504] T 22f6090263a14cde98886475acecba83 P 26ef4abb4a48496e890d727f5e551105 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26ef4abb4a48496e890d727f5e551105" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 43373 } } peers { permanent_uuid: "abce6d4107764e18b0bccc5d2799434c" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37949 } } peers { permanent_uuid: "75040f489b5f42d3b5a1e3f264acdd2b" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 35509 } }
I20260928 21:54:09.861600 5090 leader_election.cc:288] T 22f6090263a14cde98886475acecba83 P 26ef4abb4a48496e890d727f5e551105 [CANDIDATE]: Term 1 election: Requested vote from peers abce6d4107764e18b0bccc5d2799434c (127.2.182.66:37949), 75040f489b5f42d3b5a1e3f264acdd2b (127.2.182.67:35509)
I20260928 21:54:09.861799 4795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22f6090263a14cde98886475acecba83" candidate_uuid: "26ef4abb4a48496e890d727f5e551105" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abce6d4107764e18b0bccc5d2799434c"
I20260928 21:54:09.861871 4795 raft_consensus.cc:3037] T 22f6090263a14cde98886475acecba83 P abce6d4107764e18b0bccc5d2799434c [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:54:09.861830 4935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22f6090263a14cde98886475acecba83" candidate_uuid: "26ef4abb4a48496e890d727f5e551105" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75040f489b5f42d3b5a1e3f264acdd2b"
I20260928 21:54:09.861903 4935 raft_consensus.cc:3037] T 22f6090263a14cde98886475acecba83 P 75040f489b5f42d3b5a1e3f264acdd2b [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:54:09.862569 4795 raft_consensus.cc:2454] T 22f6090263a14cde98886475acecba83 P abce6d4107764e18b0bccc5d2799434c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26ef4abb4a48496e890d727f5e551105 in term 1.
I20260928 21:54:09.862572 4935 raft_consensus.cc:2454] T 22f6090263a14cde98886475acecba83 P 75040f489b5f42d3b5a1e3f264acdd2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26ef4abb4a48496e890d727f5e551105 in term 1.
I20260928 21:54:09.862748 4589 leader_election.cc:302] T 22f6090263a14cde98886475acecba83 P 26ef4abb4a48496e890d727f5e551105 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26ef4abb4a48496e890d727f5e551105, 75040f489b5f42d3b5a1e3f264acdd2b; no voters:
I20260928 21:54:09.862851 5090 raft_consensus.cc:2781] T 22f6090263a14cde98886475acecba83 P 26ef4abb4a48496e890d727f5e551105 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:54:09.863003 5090 raft_consensus.cc:686] T 22f6090263a14cde98886475acecba83 P 26ef4abb4a48496e890d727f5e551105 [term 1 LEADER]: Becoming Leader. State: Replica: 26ef4abb4a48496e890d727f5e551105, State: Running, Role: LEADER
I20260928 21:54:09.863070 5090 consensus_queue.cc:237] T 22f6090263a14cde98886475acecba83 P 26ef4abb4a48496e890d727f5e551105 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26ef4abb4a48496e890d727f5e551105" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 43373 } } peers { permanent_uuid: "abce6d4107764e18b0bccc5d2799434c" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37949 } } peers { permanent_uuid: "75040f489b5f42d3b5a1e3f264acdd2b" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 35509 } }
I20260928 21:54:09.863963 4496 catalog_manager.cc:5495] T 22f6090263a14cde98886475acecba83 P 26ef4abb4a48496e890d727f5e551105 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26ef4abb4a48496e890d727f5e551105 (127.2.182.65). New cstate: current_term: 1 leader_uuid: "26ef4abb4a48496e890d727f5e551105" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26ef4abb4a48496e890d727f5e551105" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 43373 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "abce6d4107764e18b0bccc5d2799434c" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75040f489b5f42d3b5a1e3f264acdd2b" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 35509 } health_report { overall_health: UNKNOWN } } }
I20260928 21:54:09.881747 5099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:09.879144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33454 (local address 127.2.182.65:43373)
0928 21:54:09.879301 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:09.879303 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:09.879309 (+ 6us) server_negotiation.cc:407] Connection header received
0928 21:54:09.879377 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:09.879378 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:09.879402 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:09.879489 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:54:09.879616 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:09.880185 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:09.880920 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:09.881143 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:09.881196 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:09.881373 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:54:09.881489 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:54:09.881551 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:09.881593 (+ 42us) server_negotiation.cc:299] Negotiation successful
0928 21:54:09.881614 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":85,"threads_started":1}
I20260928 21:54:09.887138 5094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:09.884599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.65:44207 (local address 127.2.182.66:37949)
0928 21:54:09.884704 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:09.884706 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:09.884791 (+ 85us) server_negotiation.cc:407] Connection header received
0928 21:54:09.884856 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:09.884856 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:09.884869 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:09.884907 (+ 38us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:54:09.885025 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:09.885482 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:09.886369 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:09.886560 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:09.886615 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:09.886867 (+ 252us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:54:09.886895 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:54:09.886941 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:09.887050 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 21:54:09.887071 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 21:54:09.887177 5102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:09.884450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.66:37949 (local address 127.2.182.65:44207)
0928 21:54:09.884738 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:09.884749 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:09.884804 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:09.884919 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:09.884919 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:09.884920 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 21:54:09.884988 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:09.884989 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:09.885565 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:09.885566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:09.886285 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:09.886288 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:09.886639 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:09.886640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:09.886767 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:09.886784 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 21:54:09.886983 (+ 199us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 21:54:09.886985 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 21:54:09.887042 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 21:54:09.887096 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":86,"threads_started":1}
I20260928 21:54:09.889217 4935 raft_consensus.cc:1261] T 22f6090263a14cde98886475acecba83 P 75040f489b5f42d3b5a1e3f264acdd2b [term 1 FOLLOWER]: Refusing update from remote peer 26ef4abb4a48496e890d727f5e551105: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:54:09.889333 4795 raft_consensus.cc:1261] T 22f6090263a14cde98886475acecba83 P abce6d4107764e18b0bccc5d2799434c [term 1 FOLLOWER]: Refusing update from remote peer 26ef4abb4a48496e890d727f5e551105: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:54:09.889509 5096 consensus_queue.cc:1035] T 22f6090263a14cde98886475acecba83 P 26ef4abb4a48496e890d727f5e551105 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75040f489b5f42d3b5a1e3f264acdd2b" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:54:09.889627 5090 consensus_queue.cc:1035] T 22f6090263a14cde98886475acecba83 P 26ef4abb4a48496e890d727f5e551105 [LEADER]: Connected to new peer: Peer: permanent_uuid: "abce6d4107764e18b0bccc5d2799434c" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:54:09.897533 5106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:09.894680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.65:43373 (local address 127.2.182.66:58153)
0928 21:54:09.894906 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:09.894918 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:09.894968 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:09.895064 (+ 96us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:09.895065 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:09.895066 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 21:54:09.895113 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:09.895115 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:09.895841 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:09.895842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:09.896657 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:09.896659 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:09.897105 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:09.897107 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:09.897159 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:09.897165 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 21:54:09.897377 (+ 212us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 21:54:09.897379 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 21:54:09.897419 (+ 40us) client_negotiation.cc:240] Negotiation successful
0928 21:54:09.897462 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":87,"threads_started":1}
I20260928 21:54:09.897547 5099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:09.894881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.66:58153 (local address 127.2.182.65:43373)
0928 21:54:09.894943 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:09.894945 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:09.894954 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:54:09.895011 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:09.895012 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:09.895021 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:09.895061 (+ 40us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:54:09.895153 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:09.895683 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:09.896793 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:09.897004 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:09.897067 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:09.897277 (+ 210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:54:09.897310 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:54:09.897361 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:09.897462 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 21:54:09.897487 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 21:54:09.907277 4496 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42860:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:54:09.907434 4496 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42860:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:54:09.908119 4496 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 39ad015d374848d084369d0f182d3f51: Sending DeleteTablet for 3 replicas of tablet 22f6090263a14cde98886475acecba83
I20260928 21:54:09.908384 4915 tablet_service.cc:1514] Processing DeleteTablet for tablet 22f6090263a14cde98886475acecba83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:54:09 UTC) from {username='kudu', principal='kudu/127.2.182.126@KRBTEST.COM'} at 127.0.0.1:38864
I20260928 21:54:09.908435 4628 tablet_service.cc:1514] Processing DeleteTablet for tablet 22f6090263a14cde98886475acecba83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:54:09 UTC) from {username='kudu', principal='kudu/127.2.182.126@KRBTEST.COM'} at 127.0.0.1:33448
I20260928 21:54:09.908479 5074 tablet_replica.cc:330] T 22f6090263a14cde98886475acecba83 P 75040f489b5f42d3b5a1e3f264acdd2b: stopping tablet replica
I20260928 21:54:09.908551 5074 raft_consensus.cc:2229] T 22f6090263a14cde98886475acecba83 P 75040f489b5f42d3b5a1e3f264acdd2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:54:09.908599 5074 raft_consensus.cc:2258] T 22f6090263a14cde98886475acecba83 P 75040f489b5f42d3b5a1e3f264acdd2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:54:09.908676 5073 tablet_replica.cc:330] T 22f6090263a14cde98886475acecba83 P 26ef4abb4a48496e890d727f5e551105: stopping tablet replica
I20260928 21:54:09.908731 5073 raft_consensus.cc:2229] T 22f6090263a14cde98886475acecba83 P 26ef4abb4a48496e890d727f5e551105 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:54:09.908818 5073 raft_consensus.cc:2258] T 22f6090263a14cde98886475acecba83 P 26ef4abb4a48496e890d727f5e551105 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:54:09.909097 4774 tablet_service.cc:1514] Processing DeleteTablet for tablet 22f6090263a14cde98886475acecba83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:54:09 UTC) from {username='kudu', principal='kudu/127.2.182.126@KRBTEST.COM'} at 127.0.0.1:44464
I20260928 21:54:09.909200 5075 tablet_replica.cc:330] T 22f6090263a14cde98886475acecba83 P abce6d4107764e18b0bccc5d2799434c: stopping tablet replica
I20260928 21:54:09.909250 5075 raft_consensus.cc:2229] T 22f6090263a14cde98886475acecba83 P abce6d4107764e18b0bccc5d2799434c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:54:09.909253 5074 ts_tablet_manager.cc:1905] T 22f6090263a14cde98886475acecba83 P 75040f489b5f42d3b5a1e3f264acdd2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:54:09.909299 5075 raft_consensus.cc:2258] T 22f6090263a14cde98886475acecba83 P abce6d4107764e18b0bccc5d2799434c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:54:09.910596 5073 ts_tablet_manager.cc:1905] T 22f6090263a14cde98886475acecba83 P 26ef4abb4a48496e890d727f5e551105: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:54:09.910635 5075 ts_tablet_manager.cc:1905] T 22f6090263a14cde98886475acecba83 P abce6d4107764e18b0bccc5d2799434c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:54:09.911468 4496 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42860:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:54:09.911659 4496 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:54:09.912169 5075 ts_tablet_manager.cc:1918] T 22f6090263a14cde98886475acecba83 P abce6d4107764e18b0bccc5d2799434c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:54:09.912227 5075 log.cc:1196] T 22f6090263a14cde98886475acecba83 P abce6d4107764e18b0bccc5d2799434c: Deleting WAL directory at /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/minicluster-data/ts-1/wal/wals/22f6090263a14cde98886475acecba83
I20260928 21:54:09.912243 5074 ts_tablet_manager.cc:1918] T 22f6090263a14cde98886475acecba83 P 75040f489b5f42d3b5a1e3f264acdd2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:54:09.912303 5074 log.cc:1196] T 22f6090263a14cde98886475acecba83 P 75040f489b5f42d3b5a1e3f264acdd2b: Deleting WAL directory at /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/minicluster-data/ts-2/wal/wals/22f6090263a14cde98886475acecba83
I20260928 21:54:09.912423 5075 ts_tablet_manager.cc:1939] T 22f6090263a14cde98886475acecba83 P abce6d4107764e18b0bccc5d2799434c: Deleting consensus metadata
I20260928 21:54:09.912516 5074 ts_tablet_manager.cc:1939] T 22f6090263a14cde98886475acecba83 P 75040f489b5f42d3b5a1e3f264acdd2b: Deleting consensus metadata
I20260928 21:54:09.912582 5073 ts_tablet_manager.cc:1918] T 22f6090263a14cde98886475acecba83 P 26ef4abb4a48496e890d727f5e551105: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:54:09.912627 5073 log.cc:1196] T 22f6090263a14cde98886475acecba83 P 26ef4abb4a48496e890d727f5e551105: Deleting WAL directory at /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/minicluster-data/ts-0/wal/wals/22f6090263a14cde98886475acecba83
I20260928 21:54:09.912843 5073 ts_tablet_manager.cc:1939] T 22f6090263a14cde98886475acecba83 P 26ef4abb4a48496e890d727f5e551105: Deleting consensus metadata
I20260928 21:54:09.912897 4485 catalog_manager.cc:4841] TS 75040f489b5f42d3b5a1e3f264acdd2b (127.2.182.67:35509): tablet 22f6090263a14cde98886475acecba83 (table test-table [id=7d5b01c9270647c9b5590d40a033f479]) successfully deleted
I20260928 21:54:09.913172 4484 catalog_manager.cc:4841] TS 26ef4abb4a48496e890d727f5e551105 (127.2.182.65:43373): tablet 22f6090263a14cde98886475acecba83 (table test-table [id=7d5b01c9270647c9b5590d40a033f479]) successfully deleted
I20260928 21:54:09.913265 4484 catalog_manager.cc:4841] TS abce6d4107764e18b0bccc5d2799434c (127.2.182.66:37949): tablet 22f6090263a14cde98886475acecba83 (table test-table [id=7d5b01c9270647c9b5590d40a033f479]) successfully deleted
I20260928 21:54:09.913501 4628 tablet_service.cc:1467] Processing CreateTablet for tablet bbe89d86b7b34702afb39c401adb7fb7 (DEFAULT_TABLE table=test-table [id=b2290816cba6417f994493cdb6d876a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:54:09.913522 4774 tablet_service.cc:1467] Processing CreateTablet for tablet bbe89d86b7b34702afb39c401adb7fb7 (DEFAULT_TABLE table=test-table [id=b2290816cba6417f994493cdb6d876a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:54:09.913633 4628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbe89d86b7b34702afb39c401adb7fb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:54:09.913654 4774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbe89d86b7b34702afb39c401adb7fb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:54:09.914599 5086 tablet_bootstrap.cc:492] T bbe89d86b7b34702afb39c401adb7fb7 P abce6d4107764e18b0bccc5d2799434c: Bootstrap starting.
I20260928 21:54:09.914856 5084 tablet_bootstrap.cc:492] T bbe89d86b7b34702afb39c401adb7fb7 P 26ef4abb4a48496e890d727f5e551105: Bootstrap starting.
I20260928 21:54:09.914909 5086 tablet_bootstrap.cc:654] T bbe89d86b7b34702afb39c401adb7fb7 P abce6d4107764e18b0bccc5d2799434c: Neither blocks nor log segments found. Creating new log.
I20260928 21:54:09.915192 4915 tablet_service.cc:1467] Processing CreateTablet for tablet bbe89d86b7b34702afb39c401adb7fb7 (DEFAULT_TABLE table=test-table [id=b2290816cba6417f994493cdb6d876a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:54:09.915248 5084 tablet_bootstrap.cc:654] T bbe89d86b7b34702afb39c401adb7fb7 P 26ef4abb4a48496e890d727f5e551105: Neither blocks nor log segments found. Creating new log.
I20260928 21:54:09.915308 4915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbe89d86b7b34702afb39c401adb7fb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:54:09.916229 5085 tablet_bootstrap.cc:492] T bbe89d86b7b34702afb39c401adb7fb7 P 75040f489b5f42d3b5a1e3f264acdd2b: Bootstrap starting.
I20260928 21:54:09.916357 5084 tablet_bootstrap.cc:492] T bbe89d86b7b34702afb39c401adb7fb7 P 26ef4abb4a48496e890d727f5e551105: No bootstrap required, opened a new log
I20260928 21:54:09.916404 5084 ts_tablet_manager.cc:1397] T bbe89d86b7b34702afb39c401adb7fb7 P 26ef4abb4a48496e890d727f5e551105: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 21:54:09.916576 5085 tablet_bootstrap.cc:654] T bbe89d86b7b34702afb39c401adb7fb7 P 75040f489b5f42d3b5a1e3f264acdd2b: Neither blocks nor log segments found. Creating new log.
I20260928 21:54:09.916561 5084 raft_consensus.cc:348] T bbe89d86b7b34702afb39c401adb7fb7 P 26ef4abb4a48496e890d727f5e551105 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26ef4abb4a48496e890d727f5e551105" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 43373 } } peers { permanent_uuid: "abce6d4107764e18b0bccc5d2799434c" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37949 } } peers { permanent_uuid: "75040f489b5f42d3b5a1e3f264acdd2b" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 35509 } }
I20260928 21:54:09.916630 5084 raft_consensus.cc:374] T bbe89d86b7b34702afb39c401adb7fb7 P 26ef4abb4a48496e890d727f5e551105 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:54:09.916646 5084 raft_consensus.cc:729] T bbe89d86b7b34702afb39c401adb7fb7 P 26ef4abb4a48496e890d727f5e551105 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26ef4abb4a48496e890d727f5e551105, State: Initialized, Role: FOLLOWER
I20260928 21:54:09.916687 5084 consensus_queue.cc:260] T bbe89d86b7b34702afb39c401adb7fb7 P 26ef4abb4a48496e890d727f5e551105 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26ef4abb4a48496e890d727f5e551105" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 43373 } } peers { permanent_uuid: "abce6d4107764e18b0bccc5d2799434c" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37949 } } peers { permanent_uuid: "75040f489b5f42d3b5a1e3f264acdd2b" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 35509 } }
I20260928 21:54:09.916813 5084 ts_tablet_manager.cc:1428] T bbe89d86b7b34702afb39c401adb7fb7 P 26ef4abb4a48496e890d727f5e551105: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:09.916966 5086 tablet_bootstrap.cc:492] T bbe89d86b7b34702afb39c401adb7fb7 P abce6d4107764e18b0bccc5d2799434c: No bootstrap required, opened a new log
I20260928 21:54:09.917019 5086 ts_tablet_manager.cc:1397] T bbe89d86b7b34702afb39c401adb7fb7 P abce6d4107764e18b0bccc5d2799434c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:54:09.917181 5085 tablet_bootstrap.cc:492] T bbe89d86b7b34702afb39c401adb7fb7 P 75040f489b5f42d3b5a1e3f264acdd2b: No bootstrap required, opened a new log
I20260928 21:54:09.917128 5086 raft_consensus.cc:348] T bbe89d86b7b34702afb39c401adb7fb7 P abce6d4107764e18b0bccc5d2799434c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26ef4abb4a48496e890d727f5e551105" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 43373 } } peers { permanent_uuid: "abce6d4107764e18b0bccc5d2799434c" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37949 } } peers { permanent_uuid: "75040f489b5f42d3b5a1e3f264acdd2b" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 35509 } }
I20260928 21:54:09.917205 5086 raft_consensus.cc:374] T bbe89d86b7b34702afb39c401adb7fb7 P abce6d4107764e18b0bccc5d2799434c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:54:09.917217 5085 ts_tablet_manager.cc:1397] T bbe89d86b7b34702afb39c401adb7fb7 P 75040f489b5f42d3b5a1e3f264acdd2b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:54:09.917225 5086 raft_consensus.cc:729] T bbe89d86b7b34702afb39c401adb7fb7 P abce6d4107764e18b0bccc5d2799434c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abce6d4107764e18b0bccc5d2799434c, State: Initialized, Role: FOLLOWER
I20260928 21:54:09.917270 5086 consensus_queue.cc:260] T bbe89d86b7b34702afb39c401adb7fb7 P abce6d4107764e18b0bccc5d2799434c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26ef4abb4a48496e890d727f5e551105" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 43373 } } peers { permanent_uuid: "abce6d4107764e18b0bccc5d2799434c" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37949 } } peers { permanent_uuid: "75040f489b5f42d3b5a1e3f264acdd2b" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 35509 } }
I20260928 21:54:09.917371 5086 ts_tablet_manager.cc:1428] T bbe89d86b7b34702afb39c401adb7fb7 P abce6d4107764e18b0bccc5d2799434c: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 21:54:09.917349 5085 raft_consensus.cc:348] T bbe89d86b7b34702afb39c401adb7fb7 P 75040f489b5f42d3b5a1e3f264acdd2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26ef4abb4a48496e890d727f5e551105" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 43373 } } peers { permanent_uuid: "abce6d4107764e18b0bccc5d2799434c" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37949 } } peers { permanent_uuid: "75040f489b5f42d3b5a1e3f264acdd2b" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 35509 } }
I20260928 21:54:09.917407 5085 raft_consensus.cc:374] T bbe89d86b7b34702afb39c401adb7fb7 P 75040f489b5f42d3b5a1e3f264acdd2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:54:09.917425 5085 raft_consensus.cc:729] T bbe89d86b7b34702afb39c401adb7fb7 P 75040f489b5f42d3b5a1e3f264acdd2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75040f489b5f42d3b5a1e3f264acdd2b, State: Initialized, Role: FOLLOWER
I20260928 21:54:09.917466 5085 consensus_queue.cc:260] T bbe89d86b7b34702afb39c401adb7fb7 P 75040f489b5f42d3b5a1e3f264acdd2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26ef4abb4a48496e890d727f5e551105" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 43373 } } peers { permanent_uuid: "abce6d4107764e18b0bccc5d2799434c" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37949 } } peers { permanent_uuid: "75040f489b5f42d3b5a1e3f264acdd2b" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 35509 } }
I20260928 21:54:09.917543 5085 ts_tablet_manager.cc:1428] T bbe89d86b7b34702afb39c401adb7fb7 P 75040f489b5f42d3b5a1e3f264acdd2b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:09.968947 5091 raft_consensus.cc:482] T bbe89d86b7b34702afb39c401adb7fb7 P 75040f489b5f42d3b5a1e3f264acdd2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:54:09.969065 5091 raft_consensus.cc:504] T bbe89d86b7b34702afb39c401adb7fb7 P 75040f489b5f42d3b5a1e3f264acdd2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26ef4abb4a48496e890d727f5e551105" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 43373 } } peers { permanent_uuid: "abce6d4107764e18b0bccc5d2799434c" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37949 } } peers { permanent_uuid: "75040f489b5f42d3b5a1e3f264acdd2b" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 35509 } }
I20260928 21:54:09.969419 5091 leader_election.cc:288] T bbe89d86b7b34702afb39c401adb7fb7 P 75040f489b5f42d3b5a1e3f264acdd2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26ef4abb4a48496e890d727f5e551105 (127.2.182.65:43373), abce6d4107764e18b0bccc5d2799434c (127.2.182.66:37949)
I20260928 21:54:09.973068 5099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:09.969707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.67:51811 (local address 127.2.182.65:43373)
0928 21:54:09.969754 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:09.969756 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:09.969866 (+ 110us) server_negotiation.cc:407] Connection header received
0928 21:54:09.969978 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:09.969979 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:09.969989 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:09.970019 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:54:09.970286 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:09.970993 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:09.972331 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:09.972704 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:09.972770 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:09.972780 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:09.972940 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 21:54:09.972986 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 21:54:09.973186 5113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:09.969824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.66:37949 (local address 127.2.182.67:56561)
0928 21:54:09.969948 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:09.969956 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:09.970003 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:09.970202 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:09.970203 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:09.970206 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:54:09.970271 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:09.970273 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:09.970898 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:09.970899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:09.972186 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:09.972190 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:09.972930 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:09.972931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:09.972994 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:09.972998 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:54:09.973059 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 21:54:09.973101 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":81,"threads_started":1}
I20260928 21:54:09.973208 5094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:09.969897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.67:56561 (local address 127.2.182.66:37949)
0928 21:54:09.969939 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:09.969941 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:09.969999 (+ 58us) server_negotiation.cc:407] Connection header received
0928 21:54:09.970056 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:09.970057 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:09.970066 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:09.970136 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:54:09.970331 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:09.970811 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:09.972299 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:09.972841 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:09.972926 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:09.972936 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:09.973081 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 21:54:09.973127 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 21:54:09.973420 4795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbe89d86b7b34702afb39c401adb7fb7" candidate_uuid: "75040f489b5f42d3b5a1e3f264acdd2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abce6d4107764e18b0bccc5d2799434c" is_pre_election: true
I20260928 21:54:09.973532 5112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:09.969630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.65:43373 (local address 127.2.182.67:51811)
0928 21:54:09.969802 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:09.969814 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:09.969895 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:09.970088 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:09.970088 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:09.970091 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:54:09.970160 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:09.970174 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:09.971101 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:09.971102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:09.972268 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:09.972270 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:09.972780 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:09.972781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:09.972858 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:09.972863 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 21:54:09.973397 (+ 534us) client_negotiation.cc:240] Negotiation successful
0928 21:54:09.973440 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":90,"threads_started":1}
I20260928 21:54:09.973527 4795 raft_consensus.cc:2454] T bbe89d86b7b34702afb39c401adb7fb7 P abce6d4107764e18b0bccc5d2799434c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75040f489b5f42d3b5a1e3f264acdd2b in term 0.
I20260928 21:54:09.973784 4868 leader_election.cc:302] T bbe89d86b7b34702afb39c401adb7fb7 P 75040f489b5f42d3b5a1e3f264acdd2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75040f489b5f42d3b5a1e3f264acdd2b, abce6d4107764e18b0bccc5d2799434c; no voters:
I20260928 21:54:09.973853 4655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbe89d86b7b34702afb39c401adb7fb7" candidate_uuid: "75040f489b5f42d3b5a1e3f264acdd2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26ef4abb4a48496e890d727f5e551105" is_pre_election: true
I20260928 21:54:09.973953 4655 raft_consensus.cc:2454] T bbe89d86b7b34702afb39c401adb7fb7 P 26ef4abb4a48496e890d727f5e551105 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75040f489b5f42d3b5a1e3f264acdd2b in term 0.
I20260928 21:54:09.974051 5091 raft_consensus.cc:2781] T bbe89d86b7b34702afb39c401adb7fb7 P 75040f489b5f42d3b5a1e3f264acdd2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:54:09.974100 5091 raft_consensus.cc:482] T bbe89d86b7b34702afb39c401adb7fb7 P 75040f489b5f42d3b5a1e3f264acdd2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:54:09.974129 5091 raft_consensus.cc:3037] T bbe89d86b7b34702afb39c401adb7fb7 P 75040f489b5f42d3b5a1e3f264acdd2b [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:54:09.975116 5091 raft_consensus.cc:504] T bbe89d86b7b34702afb39c401adb7fb7 P 75040f489b5f42d3b5a1e3f264acdd2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26ef4abb4a48496e890d727f5e551105" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 43373 } } peers { permanent_uuid: "abce6d4107764e18b0bccc5d2799434c" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37949 } } peers { permanent_uuid: "75040f489b5f42d3b5a1e3f264acdd2b" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 35509 } }
I20260928 21:54:09.975268 5091 leader_election.cc:288] T bbe89d86b7b34702afb39c401adb7fb7 P 75040f489b5f42d3b5a1e3f264acdd2b [CANDIDATE]: Term 1 election: Requested vote from peers 26ef4abb4a48496e890d727f5e551105 (127.2.182.65:43373), abce6d4107764e18b0bccc5d2799434c (127.2.182.66:37949)
I20260928 21:54:09.975495 4795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbe89d86b7b34702afb39c401adb7fb7" candidate_uuid: "75040f489b5f42d3b5a1e3f264acdd2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abce6d4107764e18b0bccc5d2799434c"
I20260928 21:54:09.975585 4795 raft_consensus.cc:3037] T bbe89d86b7b34702afb39c401adb7fb7 P abce6d4107764e18b0bccc5d2799434c [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:54:09.976188 4655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbe89d86b7b34702afb39c401adb7fb7" candidate_uuid: "75040f489b5f42d3b5a1e3f264acdd2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26ef4abb4a48496e890d727f5e551105"
I20260928 21:54:09.976266 4655 raft_consensus.cc:3037] T bbe89d86b7b34702afb39c401adb7fb7 P 26ef4abb4a48496e890d727f5e551105 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:54:09.976348 4795 raft_consensus.cc:2454] T bbe89d86b7b34702afb39c401adb7fb7 P abce6d4107764e18b0bccc5d2799434c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75040f489b5f42d3b5a1e3f264acdd2b in term 1.
I20260928 21:54:09.976493 4868 leader_election.cc:302] T bbe89d86b7b34702afb39c401adb7fb7 P 75040f489b5f42d3b5a1e3f264acdd2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75040f489b5f42d3b5a1e3f264acdd2b, abce6d4107764e18b0bccc5d2799434c; no voters:
I20260928 21:54:09.977028 4655 raft_consensus.cc:2454] T bbe89d86b7b34702afb39c401adb7fb7 P 26ef4abb4a48496e890d727f5e551105 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75040f489b5f42d3b5a1e3f264acdd2b in term 1.
I20260928 21:54:09.977133 5091 raft_consensus.cc:2781] T bbe89d86b7b34702afb39c401adb7fb7 P 75040f489b5f42d3b5a1e3f264acdd2b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:54:09.977296 5091 raft_consensus.cc:686] T bbe89d86b7b34702afb39c401adb7fb7 P 75040f489b5f42d3b5a1e3f264acdd2b [term 1 LEADER]: Becoming Leader. State: Replica: 75040f489b5f42d3b5a1e3f264acdd2b, State: Running, Role: LEADER
I20260928 21:54:09.977377 5091 consensus_queue.cc:237] T bbe89d86b7b34702afb39c401adb7fb7 P 75040f489b5f42d3b5a1e3f264acdd2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26ef4abb4a48496e890d727f5e551105" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 43373 } } peers { permanent_uuid: "abce6d4107764e18b0bccc5d2799434c" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37949 } } peers { permanent_uuid: "75040f489b5f42d3b5a1e3f264acdd2b" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 35509 } }
I20260928 21:54:09.978104 4497 catalog_manager.cc:5495] T bbe89d86b7b34702afb39c401adb7fb7 P 75040f489b5f42d3b5a1e3f264acdd2b reported cstate change: term changed from 0 to 1, leader changed from <none> to 75040f489b5f42d3b5a1e3f264acdd2b (127.2.182.67). New cstate: current_term: 1 leader_uuid: "75040f489b5f42d3b5a1e3f264acdd2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26ef4abb4a48496e890d727f5e551105" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 43373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "abce6d4107764e18b0bccc5d2799434c" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75040f489b5f42d3b5a1e3f264acdd2b" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 35509 } health_report { overall_health: HEALTHY } } }
I20260928 21:54:09.996551 5095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:09.993442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38870 (local address 127.2.182.67:35509)
0928 21:54:09.993484 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:09.993486 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:09.993494 (+ 8us) server_negotiation.cc:407] Connection header received
0928 21:54:09.993533 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:09.993534 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:09.993558 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:09.993662 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:54:09.993761 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:09.994481 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:09.995419 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:09.995748 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:09.995965 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:09.996143 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:54:09.996290 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:54:09.996366 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:09.996440 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 21:54:09.996466 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 21:54:10.002512 5094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:09.999540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.67:57289 (local address 127.2.182.66:37949)
0928 21:54:09.999589 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:09.999592 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:09.999643 (+ 51us) server_negotiation.cc:407] Connection header received
0928 21:54:09.999715 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:09.999715 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:09.999724 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:09.999769 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:54:09.999904 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:10.000423 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:10.001588 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:10.001912 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:10.001974 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:10.002221 (+ 247us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:54:10.002254 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:54:10.002298 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:10.002377 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 21:54:10.002404 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 21:54:10.002564 5118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:09.999408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.66:37949 (local address 127.2.182.67:57289)
0928 21:54:09.999589 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:09.999600 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:09.999658 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:09.999775 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:09.999776 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:09.999778 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 21:54:09.999850 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:09.999851 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:10.000495 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:10.000496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:10.001486 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:10.001490 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:10.002009 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:10.002011 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:10.002081 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:10.002105 (+ 24us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 21:54:10.002320 (+ 215us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 21:54:10.002321 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 21:54:10.002373 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 21:54:10.002434 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":92,"threads_started":1}
I20260928 21:54:10.005429 4655 raft_consensus.cc:1261] T bbe89d86b7b34702afb39c401adb7fb7 P 26ef4abb4a48496e890d727f5e551105 [term 1 FOLLOWER]: Refusing update from remote peer 75040f489b5f42d3b5a1e3f264acdd2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:54:10.005429 4795 raft_consensus.cc:1261] T bbe89d86b7b34702afb39c401adb7fb7 P abce6d4107764e18b0bccc5d2799434c [term 1 FOLLOWER]: Refusing update from remote peer 75040f489b5f42d3b5a1e3f264acdd2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:54:10.005718 5114 consensus_queue.cc:1035] T bbe89d86b7b34702afb39c401adb7fb7 P 75040f489b5f42d3b5a1e3f264acdd2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "26ef4abb4a48496e890d727f5e551105" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 43373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:54:10.005893 5091 consensus_queue.cc:1035] T bbe89d86b7b34702afb39c401adb7fb7 P 75040f489b5f42d3b5a1e3f264acdd2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "abce6d4107764e18b0bccc5d2799434c" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:54:10.014622 5095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:10.011797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.66:32971 (local address 127.2.182.67:35509)
0928 21:54:10.011836 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:10.011838 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:10.011846 (+ 8us) server_negotiation.cc:407] Connection header received
0928 21:54:10.011899 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:10.011900 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:10.011907 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:10.011954 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:54:10.012083 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:10.012723 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:10.013922 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:10.014175 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:10.014237 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:10.014403 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:54:10.014440 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:54:10.014480 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:10.014541 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 21:54:10.014560 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 21:54:10.014668 5106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:10.011762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.67:35509 (local address 127.2.182.66:32971)
0928 21:54:10.011801 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:10.011812 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:10.011851 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:10.011967 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:10.011968 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:10.011969 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 21:54:10.012016 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:10.012018 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:10.012797 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:10.012798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:10.013703 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:10.013707 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:10.014259 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:10.014260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:10.014333 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:10.014342 (+ 9us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 21:54:10.014490 (+ 148us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 21:54:10.014493 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:54:10.014537 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 21:54:10.014591 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
W20260928 21:54:10.019268 5057 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 21:54:11.549082 4496 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42860:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:54:11.549192 4496 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42860:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:54:11.550419 4496 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 39ad015d374848d084369d0f182d3f51: Sending DeleteTablet for 3 replicas of tablet bbe89d86b7b34702afb39c401adb7fb7
I20260928 21:54:11.550740 4774 tablet_service.cc:1514] Processing DeleteTablet for tablet bbe89d86b7b34702afb39c401adb7fb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:54:11 UTC) from {username='kudu', principal='kudu/127.2.182.126@KRBTEST.COM'} at 127.0.0.1:44464
I20260928 21:54:11.550763 4915 tablet_service.cc:1514] Processing DeleteTablet for tablet bbe89d86b7b34702afb39c401adb7fb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:54:11 UTC) from {username='kudu', principal='kudu/127.2.182.126@KRBTEST.COM'} at 127.0.0.1:38864
I20260928 21:54:11.550912 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 4570
I20260928 21:54:11.550948 4628 tablet_service.cc:1514] Processing DeleteTablet for tablet bbe89d86b7b34702afb39c401adb7fb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:54:11 UTC) from {username='kudu', principal='kudu/127.2.182.126@KRBTEST.COM'} at 127.0.0.1:33448
I20260928 21:54:11.552004 5144 tablet_replica.cc:330] T bbe89d86b7b34702afb39c401adb7fb7 P abce6d4107764e18b0bccc5d2799434c: stopping tablet replica
I20260928 21:54:11.552084 5144 raft_consensus.cc:2229] T bbe89d86b7b34702afb39c401adb7fb7 P abce6d4107764e18b0bccc5d2799434c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:54:11.552136 5144 raft_consensus.cc:2258] T bbe89d86b7b34702afb39c401adb7fb7 P abce6d4107764e18b0bccc5d2799434c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:54:11.552382 5144 ts_tablet_manager.cc:1905] T bbe89d86b7b34702afb39c401adb7fb7 P abce6d4107764e18b0bccc5d2799434c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:54:11.553573 5145 tablet_replica.cc:330] T bbe89d86b7b34702afb39c401adb7fb7 P 75040f489b5f42d3b5a1e3f264acdd2b: stopping tablet replica
I20260928 21:54:11.553668 5145 raft_consensus.cc:2229] T bbe89d86b7b34702afb39c401adb7fb7 P 75040f489b5f42d3b5a1e3f264acdd2b [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:54:11.553784 5145 raft_consensus.cc:2258] T bbe89d86b7b34702afb39c401adb7fb7 P 75040f489b5f42d3b5a1e3f264acdd2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:54:11.553987 5144 ts_tablet_manager.cc:1918] T bbe89d86b7b34702afb39c401adb7fb7 P abce6d4107764e18b0bccc5d2799434c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 21:54:11.554035 5144 log.cc:1196] T bbe89d86b7b34702afb39c401adb7fb7 P abce6d4107764e18b0bccc5d2799434c: Deleting WAL directory at /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/minicluster-data/ts-1/wal/wals/bbe89d86b7b34702afb39c401adb7fb7
I20260928 21:54:11.554198 5144 ts_tablet_manager.cc:1939] T bbe89d86b7b34702afb39c401adb7fb7 P abce6d4107764e18b0bccc5d2799434c: Deleting consensus metadata
I20260928 21:54:11.554401 5145 ts_tablet_manager.cc:1905] T bbe89d86b7b34702afb39c401adb7fb7 P 75040f489b5f42d3b5a1e3f264acdd2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:54:11.554621 4484 catalog_manager.cc:4841] TS abce6d4107764e18b0bccc5d2799434c (127.2.182.66:37949): tablet bbe89d86b7b34702afb39c401adb7fb7 (table test-table [id=b2290816cba6417f994493cdb6d876a3]) successfully deleted
I20260928 21:54:11.555919 5145 ts_tablet_manager.cc:1918] T bbe89d86b7b34702afb39c401adb7fb7 P 75040f489b5f42d3b5a1e3f264acdd2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 21:54:11.555972 5145 log.cc:1196] T bbe89d86b7b34702afb39c401adb7fb7 P 75040f489b5f42d3b5a1e3f264acdd2b: Deleting WAL directory at /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632440055392-2777-0/minicluster-data/ts-2/wal/wals/bbe89d86b7b34702afb39c401adb7fb7
I20260928 21:54:11.556170 5145 ts_tablet_manager.cc:1939] T bbe89d86b7b34702afb39c401adb7fb7 P 75040f489b5f42d3b5a1e3f264acdd2b: Deleting consensus metadata
I20260928 21:54:11.556552 4485 catalog_manager.cc:4841] TS 75040f489b5f42d3b5a1e3f264acdd2b (127.2.182.67:35509): tablet bbe89d86b7b34702afb39c401adb7fb7 (table test-table [id=b2290816cba6417f994493cdb6d876a3]) successfully deleted
W20260928 21:54:11.557492 4484 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.182.65:43373 (error 108)
I20260928 21:54:11.557688 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 4710
I20260928 21:54:11.561450 5147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:11.557919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.65:43373 (local address 127.0.0.1:33468)
0928 21:54:11.561324 (+ 3405us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:11.561398 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.65:43373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3374,"thread_start_us":65,"threads_started":1}
W20260928 21:54:11.561569 4484 catalog_manager.cc:4584] TS 26ef4abb4a48496e890d727f5e551105 (127.2.182.65:43373): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bbe89d86b7b34702afb39c401adb7fb7: Network error: Client connection negotiation failed: client connection to 127.2.182.65:43373: connect: Connection refused (error 111)
I20260928 21:54:11.563623 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 4850
I20260928 21:54:11.569577 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 4466
2026-09-28T21:54:11Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5593 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:54:11 dist-test-slave-k6lz krb5kdc[5150](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:54:11 dist-test-slave-k6lz krb5kdc[5150](info): set up 2 sockets
Sep 28 21:54:11 dist-test-slave-k6lz krb5kdc[5150](info): commencing operation
krb5kdc: starting...
W20260928 21:54:12.543175 2777 mini_kdc.cc:121] Time spent starting KDC: real 0.953s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Sep 28 21:54:12 dist-test-slave-k6lz krb5kdc[5150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:54:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:54:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab.
Entry for principal kudu/127.2.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab.
I20260928 21:54:12.680559 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:43871
--webserver_interface=127.2.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab
--principal=kudu/127.2.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:44163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.126:43871
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:54:12.764240 5163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:54:12.764464 5163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:54:12.764487 5163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:54:12.765760 5163 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:54:12.765816 5163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:54:12.765830 5163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:54:12.765842 5163 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:54:12.765853 5163 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:54:12.767464 5163 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:44163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.126:43871
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab
--principal=kudu/127.2.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:43871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:54:12.767750 5163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:54:12.767961 5163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:54:12.772965 5169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:12.772961 5171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:12.772961 5168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:12.773370 5163 server_base.cc:1034] running on GCE node
I20260928 21:54:12.773627 5163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:12.773927 5163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:12.775177 5163 hybrid_clock.cc:648] HybridClock initialized: now 1790632452775145 us; error 44 us; skew 500 ppm
Sep 28 21:54:12 dist-test-slave-k6lz krb5kdc[5150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632452, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:54:12.782716 5163 init.cc:377] Logged in from keytab as kudu/127.2.182.126@KRBTEST.COM (short username kudu)
I20260928 21:54:12.783155 5163 webserver.cc:460] Webserver started at http://127.2.182.126:44535/ using document root <none> and password file <none>
I20260928 21:54:12.783427 5163 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:12.783478 5163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:12.783607 5163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:54:12.784828 5163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/minicluster-data/master-0/data/instance:
uuid: "a9385081efc84184987d5641f16c2320"
format_stamp: "Formatted at 2026-09-28 21:54:12 on dist-test-slave-k6lz"
server_key: "1a3e49db7f73060a983e5cf837275740"
server_key_iv: "6bc9aed5cf3974d537f2a7a5efd75f7e"
server_key_version: "encryptionkey@0"
I20260928 21:54:12.785211 5163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/minicluster-data/master-0/wal/instance:
uuid: "a9385081efc84184987d5641f16c2320"
format_stamp: "Formatted at 2026-09-28 21:54:12 on dist-test-slave-k6lz"
server_key: "1a3e49db7f73060a983e5cf837275740"
server_key_iv: "6bc9aed5cf3974d537f2a7a5efd75f7e"
server_key_version: "encryptionkey@0"
I20260928 21:54:12.786698 5163 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 21:54:12.787621 5178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:12.787837 5163 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 21:54:12.787928 5163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/minicluster-data/master-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/minicluster-data/master-0/wal
uuid: "a9385081efc84184987d5641f16c2320"
format_stamp: "Formatted at 2026-09-28 21:54:12 on dist-test-slave-k6lz"
server_key: "1a3e49db7f73060a983e5cf837275740"
server_key_iv: "6bc9aed5cf3974d537f2a7a5efd75f7e"
server_key_version: "encryptionkey@0"
I20260928 21:54:12.788002 5163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:12.804555 5163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:54:12.807188 5163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:54:12.807322 5163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:54:12.811525 5163 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.126:43871
I20260928 21:54:12.811573 5230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.126:43871 every 8 connection(s)
I20260928 21:54:12.812018 5163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
I20260928 21:54:12.812958 5231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:54:12.815752 5231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9385081efc84184987d5641f16c2320: Bootstrap starting.
I20260928 21:54:12.815840 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 5163
I20260928 21:54:12.815909 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/minicluster-data/master-0/wal/instance
I20260928 21:54:12.816084 2777 external_mini_cluster.cc:1428] Setting key 301463f155592c20b21476d21d0d7d6a
I20260928 21:54:12.816458 5231 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9385081efc84184987d5641f16c2320: Neither blocks nor log segments found. Creating new log.
I20260928 21:54:12.816788 5231 log.cc:824] T 00000000000000000000000000000000 P a9385081efc84184987d5641f16c2320: Log is configured to *not* fsync() on all Append() calls
I20260928 21:54:12.817464 5231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9385081efc84184987d5641f16c2320: No bootstrap required, opened a new log
I20260928 21:54:12.819669 5231 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9385081efc84184987d5641f16c2320 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9385081efc84184987d5641f16c2320" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43871 } }
I20260928 21:54:12.819869 5231 raft_consensus.cc:374] T 00000000000000000000000000000000 P a9385081efc84184987d5641f16c2320 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:54:12.819905 5231 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9385081efc84184987d5641f16c2320 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9385081efc84184987d5641f16c2320, State: Initialized, Role: FOLLOWER
I20260928 21:54:12.820029 5231 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9385081efc84184987d5641f16c2320 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9385081efc84184987d5641f16c2320" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43871 } }
I20260928 21:54:12.820102 5231 raft_consensus.cc:388] T 00000000000000000000000000000000 P a9385081efc84184987d5641f16c2320 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:54:12.820137 5231 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9385081efc84184987d5641f16c2320 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:54:12.820179 5231 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9385081efc84184987d5641f16c2320 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:54:12.820808 5231 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9385081efc84184987d5641f16c2320 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9385081efc84184987d5641f16c2320" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43871 } }
I20260928 21:54:12.820896 5231 leader_election.cc:302] T 00000000000000000000000000000000 P a9385081efc84184987d5641f16c2320 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9385081efc84184987d5641f16c2320; no voters:
I20260928 21:54:12.821075 5231 leader_election.cc:288] T 00000000000000000000000000000000 P a9385081efc84184987d5641f16c2320 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:54:12.821107 5236 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9385081efc84184987d5641f16c2320 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:54:12.821308 5236 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9385081efc84184987d5641f16c2320 [term 1 LEADER]: Becoming Leader. State: Replica: a9385081efc84184987d5641f16c2320, State: Running, Role: LEADER
I20260928 21:54:12.821326 5231 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9385081efc84184987d5641f16c2320 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:54:12.821396 5236 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9385081efc84184987d5641f16c2320 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9385081efc84184987d5641f16c2320" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43871 } }
I20260928 21:54:12.821722 5236 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9385081efc84184987d5641f16c2320 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a9385081efc84184987d5641f16c2320" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9385081efc84184987d5641f16c2320" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43871 } } }
I20260928 21:54:12.821794 5236 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9385081efc84184987d5641f16c2320 [sys.catalog]: This master's current role is: LEADER
I20260928 21:54:12.821924 5236 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9385081efc84184987d5641f16c2320 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9385081efc84184987d5641f16c2320. Latest consensus state: current_term: 1 leader_uuid: "a9385081efc84184987d5641f16c2320" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9385081efc84184987d5641f16c2320" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43871 } } }
I20260928 21:54:12.821979 5236 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9385081efc84184987d5641f16c2320 [sys.catalog]: This master's current role is: LEADER
I20260928 21:54:12.822822 5242 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 28 21:54:12 dist-test-slave-k6lz krb5kdc[5150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:54:12.823370 5242 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:54:12.825057 5242 catalog_manager.cc:1347] Generated new cluster ID: ac6b1e0fc7834fa8a682d43c0140b1e7
I20260928 21:54:12.825100 5242 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:54:12.827452 5234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:12.816728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46266 (local address 127.2.182.126:43871)
0928 21:54:12.816893 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:12.816903 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:12.816935 (+ 32us) server_negotiation.cc:407] Connection header received
0928 21:54:12.817367 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:12.817373 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:12.817420 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:12.817554 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:12.818410 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:12.819424 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:12.820435 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:12.820844 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:12.820973 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:12.823660 (+ 2687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:12.823664 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:12.823665 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:12.823677 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:12.825672 (+ 1995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:12.826077 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:12.826081 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:12.826081 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:12.826218 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:12.826417 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:12.826418 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:12.826418 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:12.826789 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:12.826961 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:12.827040 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 21:54:12.827186 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":62,"threads_started":1}
I20260928 21:54:12.834120 5242 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:54:12.834906 5242 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:54:12.845984 5242 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a9385081efc84184987d5641f16c2320: Generated new TSK 0
I20260928 21:54:12.846222 5242 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.182.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab.
Entry for principal kudu/127.2.182.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab.
I20260928 21:54:12.874480 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.65:0
--local_ip_for_outbound_sockets=127.2.182.65
--webserver_interface=127.2.182.65
--webserver_port=0
--tserver_master_addrs=127.2.182.126:43871
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab
--principal=kudu/127.2.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:44163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:54:12.959546 5257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:54:12.959771 5257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:54:12.959800 5257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:54:12.961069 5257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:54:12.961134 5257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:54:12.961163 5257 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.65
I20260928 21:54:12.962754 5257 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:44163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab
--principal=kudu/127.2.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.182.65
--webserver_port=0
--tserver_master_addrs=127.2.182.126:43871
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.65
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:54:12.963003 5257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:54:12.963256 5257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:54:12.968487 5262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:12.968531 5265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:12.968513 5263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:12.968844 5257 server_base.cc:1034] running on GCE node
I20260928 21:54:12.969059 5257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:12.969319 5257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:12.970497 5257 hybrid_clock.cc:648] HybridClock initialized: now 1790632452970481 us; error 29 us; skew 500 ppm
Sep 28 21:54:12 dist-test-slave-k6lz krb5kdc[5150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632452, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:54:12.973541 5257 init.cc:377] Logged in from keytab as kudu/127.2.182.65@KRBTEST.COM (short username kudu)
I20260928 21:54:12.974051 5257 webserver.cc:460] Webserver started at http://127.2.182.65:39233/ using document root <none> and password file <none>
I20260928 21:54:12.974324 5257 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:12.974383 5257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:12.974506 5257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:54:12.975656 5257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-0/data/instance:
uuid: "0009821a049e41058052e69daabdd376"
format_stamp: "Formatted at 2026-09-28 21:54:12 on dist-test-slave-k6lz"
server_key: "3bed8ab4647a66c8991499ee2b2569eb"
server_key_iv: "f687dd13bdf8b12458af40cb50f204eb"
server_key_version: "encryptionkey@0"
I20260928 21:54:12.976001 5257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-0/wal/instance:
uuid: "0009821a049e41058052e69daabdd376"
format_stamp: "Formatted at 2026-09-28 21:54:12 on dist-test-slave-k6lz"
server_key: "3bed8ab4647a66c8991499ee2b2569eb"
server_key_iv: "f687dd13bdf8b12458af40cb50f204eb"
server_key_version: "encryptionkey@0"
I20260928 21:54:12.977339 5257 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:54:12.978240 5272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:12.978456 5257 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:54:12.978550 5257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-0/wal
uuid: "0009821a049e41058052e69daabdd376"
format_stamp: "Formatted at 2026-09-28 21:54:12 on dist-test-slave-k6lz"
server_key: "3bed8ab4647a66c8991499ee2b2569eb"
server_key_iv: "f687dd13bdf8b12458af40cb50f204eb"
server_key_version: "encryptionkey@0"
I20260928 21:54:12.978618 5257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:12.996991 5257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:54:12.999230 5257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:54:12.999418 5257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:54:12.999650 5257 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:54:12.999997 5257 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:54:13.000031 5257 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:13.000054 5257 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:54:13.000077 5257 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:13.006975 5257 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.65:45661
I20260928 21:54:13.007123 5385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.65:45661 every 8 connection(s)
I20260928 21:54:13.007465 5257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
I20260928 21:54:13.010123 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 5257
I20260928 21:54:13.010283 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-0/wal/instance
I20260928 21:54:13.010491 2777 external_mini_cluster.cc:1428] Setting key 11c7a09e4e504ce2b33eb3c4010f43c1
Sep 28 21:54:13 dist-test-slave-k6lz krb5kdc[5150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632452, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.65@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:54:13.018622 5234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:13.008374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.65:49483 (local address 127.2.182.126:43871)
0928 21:54:13.008438 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:13.008441 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:13.008619 (+ 178us) server_negotiation.cc:407] Connection header received
0928 21:54:13.009163 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:13.009165 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:13.009177 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:13.009229 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:13.010224 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:13.010847 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:13.012237 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:13.012448 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:13.012541 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:13.015231 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:13.015233 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:13.015233 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:13.015241 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:13.016795 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:13.017367 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:13.017369 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:13.017369 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:13.017437 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:13.017843 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:13.017844 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:13.017844 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:13.017989 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:13.018055 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:13.018437 (+ 382us) server_negotiation.cc:299] Negotiation successful
0928 21:54:13.018547 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
WARNING: no policy specified for kudu/127.2.182.66@KRBTEST.COM; defaulting to no policy
I20260928 21:54:13.018849 5388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:13.008272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:43871 (local address 127.2.182.65:49483)
0928 21:54:13.008493 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:13.008542 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:13.009061 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:13.009303 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:13.009307 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:13.009919 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:13.010126 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:13.010131 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:13.010935 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:13.010936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:13.012047 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:13.012050 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:13.012565 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:13.012566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:13.012718 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:13.013299 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:13.013308 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:13.015093 (+ 1785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:13.016896 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:13.016906 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:13.016909 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:13.017240 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:13.017511 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:13.017512 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:13.017512 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:13.017674 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:13.018065 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:13.018073 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:13.018269 (+ 196us) client_negotiation.cc:769] Sending connection context
0928 21:54:13.018400 (+ 131us) client_negotiation.cc:240] Negotiation successful
0928 21:54:13.018585 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":91,"threads_started":1}
I20260928 21:54:13.019532 5386 heartbeater.cc:347] Connected to a master server at 127.2.182.126:43871
I20260928 21:54:13.019649 5386 heartbeater.cc:464] Registering TS with master...
I20260928 21:54:13.019889 5386 heartbeater.cc:511] Master 127.2.182.126:43871 requested a full tablet report, sending...
I20260928 21:54:13.020697 5195 ts_manager.cc:194] Registered new tserver with Master: 0009821a049e41058052e69daabdd376 (127.2.182.65:45661)
I20260928 21:54:13.021442 5195 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.65@KRBTEST.COM'} at 127.2.182.65:49483
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab.
Entry for principal kudu/127.2.182.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab.
I20260928 21:54:13.039388 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.66:0
--local_ip_for_outbound_sockets=127.2.182.66
--webserver_interface=127.2.182.66
--webserver_port=0
--tserver_master_addrs=127.2.182.126:43871
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab
--principal=kudu/127.2.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:44163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:54:13.125161 5391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:54:13.125398 5391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:54:13.125424 5391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:54:13.126739 5391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:54:13.126803 5391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:54:13.126848 5391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.66
I20260928 21:54:13.128561 5391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:44163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab
--principal=kudu/127.2.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.182.66
--webserver_port=0
--tserver_master_addrs=127.2.182.126:43871
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5391
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.66
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:54:13.128850 5391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:54:13.129105 5391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:54:13.134624 5396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:13.134734 5399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:13.134653 5397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:13.136055 5391 server_base.cc:1034] running on GCE node
I20260928 21:54:13.136289 5391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:13.136615 5391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:13.137859 5391 hybrid_clock.cc:648] HybridClock initialized: now 1790632453137779 us; error 65 us; skew 500 ppm
Sep 28 21:54:13 dist-test-slave-k6lz krb5kdc[5150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632453, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:54:13.140758 5391 init.cc:377] Logged in from keytab as kudu/127.2.182.66@KRBTEST.COM (short username kudu)
I20260928 21:54:13.142313 5391 webserver.cc:460] Webserver started at http://127.2.182.66:43119/ using document root <none> and password file <none>
I20260928 21:54:13.142581 5391 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:13.142637 5391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:13.142772 5391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:54:13.143949 5391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-1/data/instance:
uuid: "400b733618a146378519072e5b61a5df"
format_stamp: "Formatted at 2026-09-28 21:54:13 on dist-test-slave-k6lz"
server_key: "5bcd5d92347aa6fec24a2add4bbbbdf6"
server_key_iv: "660b084658232f402068554a88451cd8"
server_key_version: "encryptionkey@0"
I20260928 21:54:13.144299 5391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-1/wal/instance:
uuid: "400b733618a146378519072e5b61a5df"
format_stamp: "Formatted at 2026-09-28 21:54:13 on dist-test-slave-k6lz"
server_key: "5bcd5d92347aa6fec24a2add4bbbbdf6"
server_key_iv: "660b084658232f402068554a88451cd8"
server_key_version: "encryptionkey@0"
I20260928 21:54:13.145735 5391 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:54:13.147128 5406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:13.147431 5391 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:54:13.147523 5391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-1/wal
uuid: "400b733618a146378519072e5b61a5df"
format_stamp: "Formatted at 2026-09-28 21:54:13 on dist-test-slave-k6lz"
server_key: "5bcd5d92347aa6fec24a2add4bbbbdf6"
server_key_iv: "660b084658232f402068554a88451cd8"
server_key_version: "encryptionkey@0"
I20260928 21:54:13.147608 5391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:13.162317 5391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:54:13.165215 5391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:54:13.165347 5391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:54:13.165566 5391 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:54:13.165889 5391 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:54:13.165935 5391 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:13.165968 5391 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:54:13.165992 5391 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:13.173748 5391 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.66:45045
I20260928 21:54:13.174109 5391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
I20260928 21:54:13.174129 5519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.66:45045 every 8 connection(s)
I20260928 21:54:13.174837 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 5391
I20260928 21:54:13.174935 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-1/wal/instance
I20260928 21:54:13.175132 2777 external_mini_cluster.cc:1428] Setting key 71e777b81e508cd4e86000f7619197dc
Sep 28 21:54:13 dist-test-slave-k6lz krb5kdc[5150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632453, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.66@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.182.67@KRBTEST.COM; defaulting to no policy
I20260928 21:54:13.186407 5234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:13.175176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.66:47737 (local address 127.2.182.126:43871)
0928 21:54:13.175234 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:13.175236 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:13.175550 (+ 314us) server_negotiation.cc:407] Connection header received
0928 21:54:13.176229 (+ 679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:13.176231 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:13.176249 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:13.176316 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:13.178066 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:13.178901 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:13.180059 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:13.180423 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:13.180553 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:13.183222 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:13.183223 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:13.183224 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:13.183233 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:13.184899 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:13.185431 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:13.185433 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:13.185433 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:13.185491 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:13.185725 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:13.185726 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:13.185726 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:13.185854 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:13.185913 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:13.186204 (+ 291us) server_negotiation.cc:299] Negotiation successful
0928 21:54:13.186298 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 21:54:13.186590 5522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:13.175134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:43871 (local address 127.2.182.66:47737)
0928 21:54:13.175401 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:13.175450 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:13.176035 (+ 585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:13.176429 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:13.176434 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:13.177607 (+ 1173us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:13.177927 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:13.177935 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:13.179030 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:13.179032 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:13.179966 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:13.179970 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:13.180615 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:13.180618 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:13.180808 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:13.181539 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:13.181550 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:13.183106 (+ 1556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:13.185037 (+ 1931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:13.185049 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:13.185051 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:13.185354 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:13.185551 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:13.185552 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:13.185552 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:13.185665 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:13.185916 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:13.185922 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:13.186078 (+ 156us) client_negotiation.cc:769] Sending connection context
0928 21:54:13.186171 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 21:54:13.186353 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":159,"threads_started":1}
I20260928 21:54:13.187548 5520 heartbeater.cc:347] Connected to a master server at 127.2.182.126:43871
I20260928 21:54:13.187665 5520 heartbeater.cc:464] Registering TS with master...
I20260928 21:54:13.187855 5520 heartbeater.cc:511] Master 127.2.182.126:43871 requested a full tablet report, sending...
I20260928 21:54:13.188338 5195 ts_manager.cc:194] Registered new tserver with Master: 400b733618a146378519072e5b61a5df (127.2.182.66:45045)
I20260928 21:54:13.188934 5195 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.66@KRBTEST.COM'} at 127.2.182.66:47737
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab.
Entry for principal kudu/127.2.182.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab.
I20260928 21:54:13.208163 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.67:0
--local_ip_for_outbound_sockets=127.2.182.67
--webserver_interface=127.2.182.67
--webserver_port=0
--tserver_master_addrs=127.2.182.126:43871
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab
--principal=kudu/127.2.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:44163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:54:13.291069 5525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:54:13.291293 5525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:54:13.291319 5525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:54:13.292795 5525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:54:13.292865 5525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:54:13.292909 5525 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.67
I20260928 21:54:13.294525 5525 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:44163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab
--principal=kudu/127.2.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.182.67
--webserver_port=0
--tserver_master_addrs=127.2.182.126:43871
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5525
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.67
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:54:13.294746 5525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:54:13.294938 5525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:54:13.300200 5533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:13.300191 5531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:13.300191 5530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:13.300269 5525 server_base.cc:1034] running on GCE node
I20260928 21:54:13.300642 5525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:13.300904 5525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:13.302071 5525 hybrid_clock.cc:648] HybridClock initialized: now 1790632453302056 us; error 31 us; skew 500 ppm
Sep 28 21:54:13 dist-test-slave-k6lz krb5kdc[5150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632453, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:54:13.304798 5525 init.cc:377] Logged in from keytab as kudu/127.2.182.67@KRBTEST.COM (short username kudu)
I20260928 21:54:13.305375 5525 webserver.cc:460] Webserver started at http://127.2.182.67:39225/ using document root <none> and password file <none>
I20260928 21:54:13.305608 5525 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:13.305657 5525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:13.305781 5525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:54:13.306866 5525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-2/data/instance:
uuid: "fe5c316540834a15af7aa5f54f28bb40"
format_stamp: "Formatted at 2026-09-28 21:54:13 on dist-test-slave-k6lz"
server_key: "2c9fe6f39e82f284c55116ecc657fb56"
server_key_iv: "e0cdbcf9b57d527177b395583248ea10"
server_key_version: "encryptionkey@0"
I20260928 21:54:13.307188 5525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-2/wal/instance:
uuid: "fe5c316540834a15af7aa5f54f28bb40"
format_stamp: "Formatted at 2026-09-28 21:54:13 on dist-test-slave-k6lz"
server_key: "2c9fe6f39e82f284c55116ecc657fb56"
server_key_iv: "e0cdbcf9b57d527177b395583248ea10"
server_key_version: "encryptionkey@0"
I20260928 21:54:13.308586 5525 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:54:13.309434 5540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:13.309805 5525 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:54:13.309890 5525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-2/wal
uuid: "fe5c316540834a15af7aa5f54f28bb40"
format_stamp: "Formatted at 2026-09-28 21:54:13 on dist-test-slave-k6lz"
server_key: "2c9fe6f39e82f284c55116ecc657fb56"
server_key_iv: "e0cdbcf9b57d527177b395583248ea10"
server_key_version: "encryptionkey@0"
I20260928 21:54:13.309968 5525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:13.334973 5525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:54:13.337707 5525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:54:13.337834 5525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:54:13.338101 5525 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:54:13.338406 5525 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:54:13.338439 5525 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:13.338461 5525 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:54:13.338474 5525 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:13.344091 5525 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.67:44431
I20260928 21:54:13.344167 5653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.67:44431 every 8 connection(s)
I20260928 21:54:13.344470 5525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
Sep 28 21:54:13 dist-test-slave-k6lz krb5kdc[5150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632453, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.67@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:54:13.353713 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 5525
I20260928 21:54:13.353816 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-2/wal/instance
I20260928 21:54:13.353984 2777 external_mini_cluster.cc:1428] Setting key 06b5ccd9b4a8d8aeef7b3cc6ec7dd17c
I20260928 21:54:13.354652 5234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:13.345190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.67:52297 (local address 127.2.182.126:43871)
0928 21:54:13.345232 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:13.345234 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:13.345507 (+ 273us) server_negotiation.cc:407] Connection header received
0928 21:54:13.346018 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:13.346019 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:13.346029 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:13.346092 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:13.347072 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:13.347667 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:13.348715 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:13.348906 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:13.348962 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:13.351395 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:13.351396 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:13.351397 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:13.351404 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:13.352724 (+ 1320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:13.353363 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:13.353364 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:13.353365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:13.353582 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:13.353956 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:13.353957 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:13.353957 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:13.354086 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:13.354158 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:13.354458 (+ 300us) server_negotiation.cc:299] Negotiation successful
0928 21:54:13.354545 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 21:54:13.354840 5656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:13.345141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:43871 (local address 127.2.182.67:52297)
0928 21:54:13.345358 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:13.345400 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:13.345917 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:13.346176 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:13.346180 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:13.346758 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:13.346959 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:13.346964 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:13.347760 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:13.347761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:13.348592 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:13.348595 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:13.348997 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:13.348998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:13.349113 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:13.349753 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:13.349761 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:13.351270 (+ 1509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:13.352895 (+ 1625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:13.352903 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:13.352905 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:13.353169 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:13.353725 (+ 556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:13.353726 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:13.353727 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:13.353881 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:13.354151 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:13.354157 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:13.354329 (+ 172us) client_negotiation.cc:769] Sending connection context
0928 21:54:13.354429 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 21:54:13.354599 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":84,"threads_started":1}
I20260928 21:54:13.355545 5654 heartbeater.cc:347] Connected to a master server at 127.2.182.126:43871
I20260928 21:54:13.355651 5654 heartbeater.cc:464] Registering TS with master...
I20260928 21:54:13.355887 5654 heartbeater.cc:511] Master 127.2.182.126:43871 requested a full tablet report, sending...
I20260928 21:54:13.356345 5195 ts_manager.cc:194] Registered new tserver with Master: fe5c316540834a15af7aa5f54f28bb40 (127.2.182.67:44431)
I20260928 21:54:13.356727 5195 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.67@KRBTEST.COM'} at 127.2.182.67:52297
I20260928 21:54:13.366492 2777 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 21:54:13.378146 5234 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:54:13.377127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46270 (local address 127.2.182.126:43871)
0928 21:54:13.377233 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:13.377235 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:13.377336 (+ 101us) server_negotiation.cc:407] Connection header received
0928 21:54:13.377405 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:13.377406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:13.377416 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:13.377466 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:13.378070 (+ 604us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46270: BlockingRecv error: recv got EOF from 127.0.0.1:46270 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 21:54:13.379089 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 5257
I20260928 21:54:13.383082 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 5391
I20260928 21:54:13.387516 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 5525
I20260928 21:54:13.391607 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 5163
2026-09-28T21:54:13Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1822 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632440055392-2777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:54:13 dist-test-slave-k6lz krb5kdc[5666](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:54:13 dist-test-slave-k6lz krb5kdc[5666](info): set up 2 sockets
Sep 28 21:54:13 dist-test-slave-k6lz krb5kdc[5666](info): commencing operation
krb5kdc: starting...
W20260928 21:54:14.383141 2777 mini_kdc.cc:121] Time spent starting KDC: real 0.971s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Sep 28 21:54:14 dist-test-slave-k6lz krb5kdc[5666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:54:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:54:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab.
Entry for principal kudu/127.2.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab.
I20260928 21:54:14.516916 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632440055392-2777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632440055392-2777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632440055392-2777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:44047
--webserver_interface=127.2.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab
--principal=kudu/127.2.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:34793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.126:44047
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:54:14.599197 5679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:54:14.599450 5679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:54:14.599473 5679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:54:14.600768 5679 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:54:14.600823 5679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:54:14.600837 5679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:54:14.600849 5679 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:54:14.600860 5679 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:54:14.602509 5679 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:34793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632440055392-2777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632440055392-2777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.126:44047
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab
--principal=kudu/127.2.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:44047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5679
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632440055392-2777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:54:14.602792 5679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:54:14.603020 5679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:54:14.608109 5685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:14.608126 5684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:14.608331 5679 server_base.cc:1034] running on GCE node
W20260928 21:54:14.608109 5687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:14.608619 5679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:14.608930 5679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:14.610085 5679 hybrid_clock.cc:648] HybridClock initialized: now 1790632454610049 us; error 57 us; skew 500 ppm
Sep 28 21:54:14 dist-test-slave-k6lz krb5kdc[5666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632454, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:54:14.612759 5679 init.cc:377] Logged in from keytab as kudu/127.2.182.126@KRBTEST.COM (short username kudu)
I20260928 21:54:14.613183 5679 webserver.cc:460] Webserver started at http://127.2.182.126:36559/ using document root <none> and password file <none>
I20260928 21:54:14.613376 5679 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:14.613416 5679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:14.613525 5679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:54:14.614596 5679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632440055392-2777-0/minicluster-data/master-0/data/instance:
uuid: "f5cccab3e65c4457bafdc570dfe345cb"
format_stamp: "Formatted at 2026-09-28 21:54:14 on dist-test-slave-k6lz"
server_key: "fb9df04916e67aca7d07617d1e486b3a"
server_key_iv: "3f631239274bbffe73c90b014f57b97b"
server_key_version: "encryptionkey@0"
I20260928 21:54:14.614897 5679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632440055392-2777-0/minicluster-data/master-0/wal/instance:
uuid: "f5cccab3e65c4457bafdc570dfe345cb"
format_stamp: "Formatted at 2026-09-28 21:54:14 on dist-test-slave-k6lz"
server_key: "fb9df04916e67aca7d07617d1e486b3a"
server_key_iv: "3f631239274bbffe73c90b014f57b97b"
server_key_version: "encryptionkey@0"
I20260928 21:54:14.616271 5679 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 21:54:14.617120 5694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:14.617416 5679 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:54:14.617525 5679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632440055392-2777-0/minicluster-data/master-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632440055392-2777-0/minicluster-data/master-0/wal
uuid: "f5cccab3e65c4457bafdc570dfe345cb"
format_stamp: "Formatted at 2026-09-28 21:54:14 on dist-test-slave-k6lz"
server_key: "fb9df04916e67aca7d07617d1e486b3a"
server_key_iv: "3f631239274bbffe73c90b014f57b97b"
server_key_version: "encryptionkey@0"
I20260928 21:54:14.617595 5679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632440055392-2777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632440055392-2777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632440055392-2777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:14.632339 5679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:54:14.634842 5679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:54:14.634996 5679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:54:14.638890 5679 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.126:44047
I20260928 21:54:14.638922 5746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.126:44047 every 8 connection(s)
I20260928 21:54:14.639446 5679 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
I20260928 21:54:14.640203 5747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:54:14.642107 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 5679
I20260928 21:54:14.642196 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632440055392-2777-0/minicluster-data/master-0/wal/instance
I20260928 21:54:14.642378 2777 external_mini_cluster.cc:1428] Setting key d1b7da633ccc50e0572d4b5734624110
I20260928 21:54:14.642635 5747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5cccab3e65c4457bafdc570dfe345cb: Bootstrap starting.
I20260928 21:54:14.643296 5747 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f5cccab3e65c4457bafdc570dfe345cb: Neither blocks nor log segments found. Creating new log.
I20260928 21:54:14.643608 5747 log.cc:824] T 00000000000000000000000000000000 P f5cccab3e65c4457bafdc570dfe345cb: Log is configured to *not* fsync() on all Append() calls
I20260928 21:54:14.644227 5747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5cccab3e65c4457bafdc570dfe345cb: No bootstrap required, opened a new log
I20260928 21:54:14.646518 5747 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5cccab3e65c4457bafdc570dfe345cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5cccab3e65c4457bafdc570dfe345cb" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 44047 } }
I20260928 21:54:14.646697 5747 raft_consensus.cc:374] T 00000000000000000000000000000000 P f5cccab3e65c4457bafdc570dfe345cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:54:14.646732 5747 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5cccab3e65c4457bafdc570dfe345cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5cccab3e65c4457bafdc570dfe345cb, State: Initialized, Role: FOLLOWER
I20260928 21:54:14.646854 5747 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5cccab3e65c4457bafdc570dfe345cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5cccab3e65c4457bafdc570dfe345cb" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 44047 } }
I20260928 21:54:14.646922 5747 raft_consensus.cc:388] T 00000000000000000000000000000000 P f5cccab3e65c4457bafdc570dfe345cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:54:14.646957 5747 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5cccab3e65c4457bafdc570dfe345cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:54:14.647001 5747 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5cccab3e65c4457bafdc570dfe345cb [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:54:14.647710 5747 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5cccab3e65c4457bafdc570dfe345cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5cccab3e65c4457bafdc570dfe345cb" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 44047 } }
I20260928 21:54:14.647823 5747 leader_election.cc:302] T 00000000000000000000000000000000 P f5cccab3e65c4457bafdc570dfe345cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f5cccab3e65c4457bafdc570dfe345cb; no voters:
I20260928 21:54:14.647989 5747 leader_election.cc:288] T 00000000000000000000000000000000 P f5cccab3e65c4457bafdc570dfe345cb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:54:14.648048 5752 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f5cccab3e65c4457bafdc570dfe345cb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:54:14.648248 5747 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5cccab3e65c4457bafdc570dfe345cb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:54:14.648253 5752 raft_consensus.cc:686] T 00000000000000000000000000000000 P f5cccab3e65c4457bafdc570dfe345cb [term 1 LEADER]: Becoming Leader. State: Replica: f5cccab3e65c4457bafdc570dfe345cb, State: Running, Role: LEADER
I20260928 21:54:14.648403 5752 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5cccab3e65c4457bafdc570dfe345cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5cccab3e65c4457bafdc570dfe345cb" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 44047 } }
I20260928 21:54:14.648746 5752 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5cccab3e65c4457bafdc570dfe345cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5cccab3e65c4457bafdc570dfe345cb. Latest consensus state: current_term: 1 leader_uuid: "f5cccab3e65c4457bafdc570dfe345cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5cccab3e65c4457bafdc570dfe345cb" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 44047 } } }
I20260928 21:54:14.648833 5752 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5cccab3e65c4457bafdc570dfe345cb [sys.catalog]: This master's current role is: LEADER
I20260928 21:54:14.648974 5754 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5cccab3e65c4457bafdc570dfe345cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f5cccab3e65c4457bafdc570dfe345cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5cccab3e65c4457bafdc570dfe345cb" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 44047 } } }
I20260928 21:54:14.649067 5754 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5cccab3e65c4457bafdc570dfe345cb [sys.catalog]: This master's current role is: LEADER
I20260928 21:54:14.649204 5761 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 28 21:54:14 dist-test-slave-k6lz krb5kdc[5666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:54:14.649695 5761 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:54:14.651660 5761 catalog_manager.cc:1347] Generated new cluster ID: 46e0403ac352460ea426d08d1bb3f5b6
I20260928 21:54:14.651721 5761 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:54:14.653872 5750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:14.642874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51488 (local address 127.2.182.126:44047)
0928 21:54:14.643055 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:14.643065 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:14.643118 (+ 53us) server_negotiation.cc:407] Connection header received
0928 21:54:14.643610 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:14.643617 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:14.643667 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:14.643813 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:14.644430 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:14.645808 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:14.646589 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:14.647180 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:14.647293 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:14.650305 (+ 3012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:14.650309 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:14.650310 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:14.650322 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:14.652286 (+ 1964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:14.652619 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:14.652623 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:14.652623 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:14.652711 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:14.652890 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:14.652891 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:14.652891 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:14.653120 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:14.653299 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:14.653385 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 21:54:14.653520 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":85,"threads_started":1}
I20260928 21:54:14.664321 2777 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:54:14.669519 5761 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:54:14.670264 5761 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:54:14.675586 5761 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f5cccab3e65c4457bafdc570dfe345cb: Generated new TSK 0
I20260928 21:54:14.675799 5761 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:54:14.679100 5750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:14.674528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51498 (local address 127.2.182.126:44047)
0928 21:54:14.674573 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:14.674575 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:14.674688 (+ 113us) server_negotiation.cc:407] Connection header received
0928 21:54:14.674747 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:14.674747 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:14.674758 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:14.674807 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:14.675231 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:14.676423 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:14.677264 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:14.677597 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:14.677672 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:14.678516 (+ 844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:14.678519 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:14.678519 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:54:14.678770 (+ 251us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:14.678790 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:54:14.678798 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:14.678865 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:14.678978 (+ 113us) server_negotiation.cc:299] Negotiation successful
0928 21:54:14.679011 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":21}
I20260928 21:54:14.682199 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 5679
2026-09-28T21:54:14Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1287 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:54:14 dist-test-slave-k6lz krb5kdc[5780](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:54:14 dist-test-slave-k6lz krb5kdc[5780](info): set up 2 sockets
Sep 28 21:54:14 dist-test-slave-k6lz krb5kdc[5780](info): commencing operation
krb5kdc: starting...
W20260928 21:54:15.647073 2777 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Sep 28 21:54:15 dist-test-slave-k6lz krb5kdc[5780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:54:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:54:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab.
Entry for principal kudu/127.2.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab.
I20260928 21:54:15.771692 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:33005
--webserver_interface=127.2.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab
--principal=kudu/127.2.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:46189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.126:33005
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:54:15.853345 5793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:54:15.853580 5793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:54:15.853631 5793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:54:15.854873 5793 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:54:15.854920 5793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:54:15.854939 5793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:54:15.854954 5793 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:54:15.854970 5793 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:54:15.856561 5793 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:46189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.126:33005
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab
--principal=kudu/127.2.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:33005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5793
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:54:15.856820 5793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:54:15.857031 5793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:54:15.862151 5799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:15.862149 5801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:15.862175 5793 server_base.cc:1034] running on GCE node
W20260928 21:54:15.862121 5798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:15.862610 5793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:15.862949 5793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:15.864107 5793 hybrid_clock.cc:648] HybridClock initialized: now 1790632455864092 us; error 31 us; skew 500 ppm
Sep 28 21:54:15 dist-test-slave-k6lz krb5kdc[5780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632455, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:54:15.866662 5793 init.cc:377] Logged in from keytab as kudu/127.2.182.126@KRBTEST.COM (short username kudu)
I20260928 21:54:15.867146 5793 webserver.cc:460] Webserver started at http://127.2.182.126:32785/ using document root <none> and password file <none>
I20260928 21:54:15.867429 5793 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:15.867486 5793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:15.867617 5793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:54:15.868662 5793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/minicluster-data/master-0/data/instance:
uuid: "502950acdfa1437ea22a58f221c36fab"
format_stamp: "Formatted at 2026-09-28 21:54:15 on dist-test-slave-k6lz"
server_key: "daf8b19ba76a172ea992a5e816176410"
server_key_iv: "bcceaf82e2ff5eaf915c4c9a93e29eeb"
server_key_version: "encryptionkey@0"
I20260928 21:54:15.868991 5793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/minicluster-data/master-0/wal/instance:
uuid: "502950acdfa1437ea22a58f221c36fab"
format_stamp: "Formatted at 2026-09-28 21:54:15 on dist-test-slave-k6lz"
server_key: "daf8b19ba76a172ea992a5e816176410"
server_key_iv: "bcceaf82e2ff5eaf915c4c9a93e29eeb"
server_key_version: "encryptionkey@0"
I20260928 21:54:15.870442 5793 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:54:15.871320 5808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:15.871558 5793 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:54:15.871651 5793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/minicluster-data/master-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/minicluster-data/master-0/wal
uuid: "502950acdfa1437ea22a58f221c36fab"
format_stamp: "Formatted at 2026-09-28 21:54:15 on dist-test-slave-k6lz"
server_key: "daf8b19ba76a172ea992a5e816176410"
server_key_iv: "bcceaf82e2ff5eaf915c4c9a93e29eeb"
server_key_version: "encryptionkey@0"
I20260928 21:54:15.871729 5793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:15.881247 5793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:54:15.883370 5793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:54:15.883524 5793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:54:15.887486 5793 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.126:33005
I20260928 21:54:15.887566 5860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.126:33005 every 8 connection(s)
I20260928 21:54:15.887882 5793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
I20260928 21:54:15.888876 5861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:54:15.891088 5861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 502950acdfa1437ea22a58f221c36fab: Bootstrap starting.
I20260928 21:54:15.891654 5861 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 502950acdfa1437ea22a58f221c36fab: Neither blocks nor log segments found. Creating new log.
I20260928 21:54:15.891911 5861 log.cc:824] T 00000000000000000000000000000000 P 502950acdfa1437ea22a58f221c36fab: Log is configured to *not* fsync() on all Append() calls
I20260928 21:54:15.892552 5861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 502950acdfa1437ea22a58f221c36fab: No bootstrap required, opened a new log
I20260928 21:54:15.894392 5861 raft_consensus.cc:348] T 00000000000000000000000000000000 P 502950acdfa1437ea22a58f221c36fab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "502950acdfa1437ea22a58f221c36fab" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 33005 } }
I20260928 21:54:15.894533 5861 raft_consensus.cc:374] T 00000000000000000000000000000000 P 502950acdfa1437ea22a58f221c36fab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:54:15.894555 5861 raft_consensus.cc:729] T 00000000000000000000000000000000 P 502950acdfa1437ea22a58f221c36fab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 502950acdfa1437ea22a58f221c36fab, State: Initialized, Role: FOLLOWER
I20260928 21:54:15.894697 5861 consensus_queue.cc:260] T 00000000000000000000000000000000 P 502950acdfa1437ea22a58f221c36fab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "502950acdfa1437ea22a58f221c36fab" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 33005 } }
I20260928 21:54:15.894757 5861 raft_consensus.cc:388] T 00000000000000000000000000000000 P 502950acdfa1437ea22a58f221c36fab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:54:15.894788 5861 raft_consensus.cc:482] T 00000000000000000000000000000000 P 502950acdfa1437ea22a58f221c36fab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:54:15.894832 5861 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 502950acdfa1437ea22a58f221c36fab [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:54:15.895748 5861 raft_consensus.cc:504] T 00000000000000000000000000000000 P 502950acdfa1437ea22a58f221c36fab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "502950acdfa1437ea22a58f221c36fab" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 33005 } }
I20260928 21:54:15.895839 5861 leader_election.cc:302] T 00000000000000000000000000000000 P 502950acdfa1437ea22a58f221c36fab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 502950acdfa1437ea22a58f221c36fab; no voters:
I20260928 21:54:15.895985 5861 leader_election.cc:288] T 00000000000000000000000000000000 P 502950acdfa1437ea22a58f221c36fab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:54:15.896023 5864 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 502950acdfa1437ea22a58f221c36fab [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:54:15.896178 5864 raft_consensus.cc:686] T 00000000000000000000000000000000 P 502950acdfa1437ea22a58f221c36fab [term 1 LEADER]: Becoming Leader. State: Replica: 502950acdfa1437ea22a58f221c36fab, State: Running, Role: LEADER
I20260928 21:54:15.896224 5861 sys_catalog.cc:564] T 00000000000000000000000000000000 P 502950acdfa1437ea22a58f221c36fab [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:54:15.896266 5864 consensus_queue.cc:237] T 00000000000000000000000000000000 P 502950acdfa1437ea22a58f221c36fab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "502950acdfa1437ea22a58f221c36fab" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 33005 } }
I20260928 21:54:15.896600 5865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 502950acdfa1437ea22a58f221c36fab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "502950acdfa1437ea22a58f221c36fab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "502950acdfa1437ea22a58f221c36fab" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 33005 } } }
I20260928 21:54:15.896687 5865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 502950acdfa1437ea22a58f221c36fab [sys.catalog]: This master's current role is: LEADER
I20260928 21:54:15.896739 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 5793
I20260928 21:54:15.896842 5866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 502950acdfa1437ea22a58f221c36fab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 502950acdfa1437ea22a58f221c36fab. Latest consensus state: current_term: 1 leader_uuid: "502950acdfa1437ea22a58f221c36fab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "502950acdfa1437ea22a58f221c36fab" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 33005 } } }
I20260928 21:54:15.896898 5866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 502950acdfa1437ea22a58f221c36fab [sys.catalog]: This master's current role is: LEADER
I20260928 21:54:15.896930 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/minicluster-data/master-0/wal/instance
I20260928 21:54:15.897120 2777 external_mini_cluster.cc:1428] Setting key f0d29bb18d403d0483b88fc23c3d4e3a
W20260928 21:54:15.898044 5880 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 502950acdfa1437ea22a58f221c36fab: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:54:15.898101 5880 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:54:15.898164 5871 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:54:15.898341 5871 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:54:15.899789 5871 catalog_manager.cc:1347] Generated new cluster ID: 85ca67f524fe44ebadd0feec888ceb61
I20260928 21:54:15.899840 5871 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 21:54:15 dist-test-slave-k6lz krb5kdc[5780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:54:15.907569 5882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:15.897702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53688 (local address 127.2.182.126:33005)
0928 21:54:15.897836 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:15.897846 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:15.897977 (+ 131us) server_negotiation.cc:407] Connection header received
0928 21:54:15.898116 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:15.898123 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:15.898183 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:15.898339 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:15.899215 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:15.900275 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:15.901004 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:15.901424 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:15.901515 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:15.903746 (+ 2231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:15.903752 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:15.903752 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:15.903764 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:15.905735 (+ 1971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:15.906060 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:15.906063 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:15.906064 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:15.906160 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:15.906498 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:15.906499 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:15.906499 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:15.906770 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:15.907006 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:15.907137 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 21:54:15.907380 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":77,"threads_started":1}
I20260928 21:54:15.913864 5871 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:54:15.914595 5871 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:54:15.928386 5871 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 502950acdfa1437ea22a58f221c36fab: Generated new TSK 0
I20260928 21:54:15.928611 5871 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.182.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab.
Entry for principal kudu/127.2.182.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab.
I20260928 21:54:15.964061 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.65:0
--local_ip_for_outbound_sockets=127.2.182.65
--webserver_interface=127.2.182.65
--webserver_port=0
--tserver_master_addrs=127.2.182.126:33005
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab
--principal=kudu/127.2.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:46189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:54:16.052667 5887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:54:16.052891 5887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:54:16.052917 5887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:54:16.054088 5887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:54:16.054132 5887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:54:16.054163 5887 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.65
I20260928 21:54:16.055668 5887 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:46189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab
--principal=kudu/127.2.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.182.65
--webserver_port=0
--tserver_master_addrs=127.2.182.126:33005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5887
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.65
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:54:16.055867 5887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:54:16.056051 5887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:54:16.061189 5892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:16.061174 5895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:16.061175 5893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:16.061439 5887 server_base.cc:1034] running on GCE node
I20260928 21:54:16.061630 5887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:16.061869 5887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:16.063021 5887 hybrid_clock.cc:648] HybridClock initialized: now 1790632456063009 us; error 28 us; skew 500 ppm
Sep 28 21:54:16 dist-test-slave-k6lz krb5kdc[5780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632456, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:54:16.065752 5887 init.cc:377] Logged in from keytab as kudu/127.2.182.65@KRBTEST.COM (short username kudu)
I20260928 21:54:16.066242 5887 webserver.cc:460] Webserver started at http://127.2.182.65:42417/ using document root <none> and password file <none>
I20260928 21:54:16.066520 5887 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:16.066579 5887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:16.066679 5887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:54:16.067920 5887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-0/data/instance:
uuid: "e0d3413d11cd4629a743fa8385d25ef4"
format_stamp: "Formatted at 2026-09-28 21:54:16 on dist-test-slave-k6lz"
server_key: "cfa92981f9a66610ed570ae8f9e1bee1"
server_key_iv: "129346ff38b6456a07dbc406e8637715"
server_key_version: "encryptionkey@0"
I20260928 21:54:16.068238 5887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-0/wal/instance:
uuid: "e0d3413d11cd4629a743fa8385d25ef4"
format_stamp: "Formatted at 2026-09-28 21:54:16 on dist-test-slave-k6lz"
server_key: "cfa92981f9a66610ed570ae8f9e1bee1"
server_key_iv: "129346ff38b6456a07dbc406e8637715"
server_key_version: "encryptionkey@0"
I20260928 21:54:16.069561 5887 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:54:16.070439 5902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:16.070596 5887 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:54:16.070679 5887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-0/wal
uuid: "e0d3413d11cd4629a743fa8385d25ef4"
format_stamp: "Formatted at 2026-09-28 21:54:16 on dist-test-slave-k6lz"
server_key: "cfa92981f9a66610ed570ae8f9e1bee1"
server_key_iv: "129346ff38b6456a07dbc406e8637715"
server_key_version: "encryptionkey@0"
I20260928 21:54:16.070741 5887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:16.081678 5887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:54:16.084050 5887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:54:16.084227 5887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:54:16.084465 5887 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:54:16.084805 5887 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:54:16.084851 5887 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:16.084882 5887 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:54:16.084898 5887 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:16.090855 5887 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.65:41875
I20260928 21:54:16.090942 6015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.65:41875 every 8 connection(s)
I20260928 21:54:16.091243 5887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
Sep 28 21:54:16 dist-test-slave-k6lz krb5kdc[5780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632456, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.65@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:54:16.099349 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 5887
I20260928 21:54:16.099442 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-0/wal/instance
I20260928 21:54:16.099625 2777 external_mini_cluster.cc:1428] Setting key e58303abd38c4c3ac77d20c2d3cb94cb
I20260928 21:54:16.101421 5882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:16.092017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.65:37451 (local address 127.2.182.126:33005)
0928 21:54:16.092092 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:16.092094 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:16.092320 (+ 226us) server_negotiation.cc:407] Connection header received
0928 21:54:16.092788 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:16.092789 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:16.092801 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:16.092867 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:16.093929 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:16.094486 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:16.095627 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:16.095847 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:16.095906 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:16.098273 (+ 2367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:16.098274 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:16.098275 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:16.098282 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:16.099951 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:16.100410 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:16.100411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:16.100412 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:16.100469 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:16.100731 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:16.100731 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:16.100731 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:16.100829 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:16.100871 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:16.101268 (+ 397us) server_negotiation.cc:299] Negotiation successful
0928 21:54:16.101352 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 21:54:16.101564 6018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:16.091951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:33005 (local address 127.2.182.65:37451)
0928 21:54:16.092163 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:16.092205 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:16.092681 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:16.092932 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:16.092935 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:16.093499 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:16.093821 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:16.093827 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:16.094591 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:16.094592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:16.095517 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:16.095545 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:16.095923 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:16.095924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:16.096058 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:16.096569 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:16.096580 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:16.098174 (+ 1594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:16.100031 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:16.100039 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:16.100042 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:16.100335 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:16.100527 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:16.100528 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:16.100528 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:16.100662 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:16.100875 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:16.100880 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:16.101049 (+ 169us) client_negotiation.cc:769] Sending connection context
0928 21:54:16.101155 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 21:54:16.101333 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":69,"threads_started":1}
I20260928 21:54:16.102156 6016 heartbeater.cc:347] Connected to a master server at 127.2.182.126:33005
I20260928 21:54:16.102254 6016 heartbeater.cc:464] Registering TS with master...
I20260928 21:54:16.102434 6016 heartbeater.cc:511] Master 127.2.182.126:33005 requested a full tablet report, sending...
I20260928 21:54:16.103005 5825 ts_manager.cc:194] Registered new tserver with Master: e0d3413d11cd4629a743fa8385d25ef4 (127.2.182.65:41875)
I20260928 21:54:16.103677 5825 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.65@KRBTEST.COM'} at 127.2.182.65:37451
WARNING: no policy specified for kudu/127.2.182.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab.
Entry for principal kudu/127.2.182.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab.
I20260928 21:54:16.126852 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.66:0
--local_ip_for_outbound_sockets=127.2.182.66
--webserver_interface=127.2.182.66
--webserver_port=0
--tserver_master_addrs=127.2.182.126:33005
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab
--principal=kudu/127.2.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:46189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:54:16.207794 6021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:54:16.208010 6021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:54:16.208045 6021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:54:16.209201 6021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:54:16.209242 6021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:54:16.209277 6021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.66
I20260928 21:54:16.210718 6021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:46189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab
--principal=kudu/127.2.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.182.66
--webserver_port=0
--tserver_master_addrs=127.2.182.126:33005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.66
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:54:16.210988 6021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:54:16.211210 6021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:54:16.216104 6029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:16.216104 6026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:16.216301 6021 server_base.cc:1034] running on GCE node
W20260928 21:54:16.216099 6027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:16.216569 6021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:16.216842 6021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:16.217993 6021 hybrid_clock.cc:648] HybridClock initialized: now 1790632456217978 us; error 33 us; skew 500 ppm
Sep 28 21:54:16 dist-test-slave-k6lz krb5kdc[5780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632456, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:54:16.220667 6021 init.cc:377] Logged in from keytab as kudu/127.2.182.66@KRBTEST.COM (short username kudu)
I20260928 21:54:16.221074 6021 webserver.cc:460] Webserver started at http://127.2.182.66:43347/ using document root <none> and password file <none>
I20260928 21:54:16.221334 6021 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:16.221383 6021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:16.221501 6021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:54:16.222556 6021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-1/data/instance:
uuid: "84f1edf83dbb4e20879e2aec1b7dea18"
format_stamp: "Formatted at 2026-09-28 21:54:16 on dist-test-slave-k6lz"
server_key: "faef123d64b05e2d28652e120bb65ff0"
server_key_iv: "b69070b08150272b19ec46dee3dbb509"
server_key_version: "encryptionkey@0"
I20260928 21:54:16.222959 6021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-1/wal/instance:
uuid: "84f1edf83dbb4e20879e2aec1b7dea18"
format_stamp: "Formatted at 2026-09-28 21:54:16 on dist-test-slave-k6lz"
server_key: "faef123d64b05e2d28652e120bb65ff0"
server_key_iv: "b69070b08150272b19ec46dee3dbb509"
server_key_version: "encryptionkey@0"
I20260928 21:54:16.224280 6021 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 21:54:16.224979 6036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:16.225129 6021 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 21:54:16.225212 6021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-1/wal
uuid: "84f1edf83dbb4e20879e2aec1b7dea18"
format_stamp: "Formatted at 2026-09-28 21:54:16 on dist-test-slave-k6lz"
server_key: "faef123d64b05e2d28652e120bb65ff0"
server_key_iv: "b69070b08150272b19ec46dee3dbb509"
server_key_version: "encryptionkey@0"
I20260928 21:54:16.225271 6021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:16.236976 6021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:54:16.239034 6021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:54:16.239174 6021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:54:16.239408 6021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:54:16.239687 6021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:54:16.239719 6021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:16.239748 6021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:54:16.239768 6021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:16.245373 6021 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.66:45639
I20260928 21:54:16.245427 6149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.66:45639 every 8 connection(s)
I20260928 21:54:16.245726 6021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
Sep 28 21:54:16 dist-test-slave-k6lz krb5kdc[5780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632456, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.66@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:54:16.251864 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 6021
I20260928 21:54:16.251960 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-1/wal/instance
I20260928 21:54:16.252131 2777 external_mini_cluster.cc:1428] Setting key d0c538174e9a7407024f0438219c75da
I20260928 21:54:16.254854 5882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:16.246412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.66:53187 (local address 127.2.182.126:33005)
0928 21:54:16.246470 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:16.246472 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:16.246649 (+ 177us) server_negotiation.cc:407] Connection header received
0928 21:54:16.247129 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:16.247130 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:16.247141 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:16.247212 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:16.248141 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:16.248668 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:16.249739 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:16.249938 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:16.249994 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:16.252280 (+ 2286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:16.252282 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:16.252282 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:16.252291 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:16.253603 (+ 1312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:16.253993 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:16.253994 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:16.253994 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:16.254045 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:16.254265 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:16.254266 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:16.254266 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:16.254374 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:16.254427 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:16.254682 (+ 255us) server_negotiation.cc:299] Negotiation successful
0928 21:54:16.254766 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 21:54:16.255034 6152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:16.246382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:33005 (local address 127.2.182.66:53187)
0928 21:54:16.246531 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:16.246574 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:16.247042 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:16.247267 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:16.247270 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:16.247841 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:16.248031 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:16.248035 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:16.248760 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:16.248760 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:16.249606 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:16.249608 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:16.250013 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:16.250014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:16.250124 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:16.250663 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:16.250672 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:16.252139 (+ 1467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:16.253700 (+ 1561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:16.253707 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:16.253710 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:16.253927 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:16.254098 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:16.254099 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:16.254099 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:16.254210 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:16.254427 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:16.254433 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:16.254570 (+ 137us) client_negotiation.cc:769] Sending connection context
0928 21:54:16.254656 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 21:54:16.254820 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":68,"threads_started":1}
I20260928 21:54:16.255573 6150 heartbeater.cc:347] Connected to a master server at 127.2.182.126:33005
I20260928 21:54:16.255656 6150 heartbeater.cc:464] Registering TS with master...
I20260928 21:54:16.255820 6150 heartbeater.cc:511] Master 127.2.182.126:33005 requested a full tablet report, sending...
I20260928 21:54:16.256174 5825 ts_manager.cc:194] Registered new tserver with Master: 84f1edf83dbb4e20879e2aec1b7dea18 (127.2.182.66:45639)
I20260928 21:54:16.256531 5825 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.66@KRBTEST.COM'} at 127.2.182.66:53187
WARNING: no policy specified for kudu/127.2.182.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab.
Entry for principal kudu/127.2.182.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab.
I20260928 21:54:16.278355 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.67:0
--local_ip_for_outbound_sockets=127.2.182.67
--webserver_interface=127.2.182.67
--webserver_port=0
--tserver_master_addrs=127.2.182.126:33005
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab
--principal=kudu/127.2.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:46189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:54:16.358410 6155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:54:16.358632 6155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:54:16.358664 6155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:54:16.359874 6155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:54:16.359923 6155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:54:16.359948 6155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.67
I20260928 21:54:16.361418 6155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:46189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab
--principal=kudu/127.2.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.182.67
--webserver_port=0
--tserver_master_addrs=127.2.182.126:33005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.67
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:54:16.361621 6155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:54:16.361816 6155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:54:16.367751 6161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:16.367820 6160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:16.367928 6155 server_base.cc:1034] running on GCE node
W20260928 21:54:16.367816 6163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:16.368193 6155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:16.368428 6155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:16.369589 6155 hybrid_clock.cc:648] HybridClock initialized: now 1790632456369573 us; error 28 us; skew 500 ppm
Sep 28 21:54:16 dist-test-slave-k6lz krb5kdc[5780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632456, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:54:16.372295 6155 init.cc:377] Logged in from keytab as kudu/127.2.182.67@KRBTEST.COM (short username kudu)
I20260928 21:54:16.372840 6155 webserver.cc:460] Webserver started at http://127.2.182.67:41059/ using document root <none> and password file <none>
I20260928 21:54:16.373044 6155 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:16.373096 6155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:16.373222 6155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:54:16.374377 6155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-2/data/instance:
uuid: "b1c3327754bc4473863b49f23c319665"
format_stamp: "Formatted at 2026-09-28 21:54:16 on dist-test-slave-k6lz"
server_key: "479965645448451e3eda4b9a25a66747"
server_key_iv: "e41231b4955068d0f39e715664a34437"
server_key_version: "encryptionkey@0"
I20260928 21:54:16.374758 6155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-2/wal/instance:
uuid: "b1c3327754bc4473863b49f23c319665"
format_stamp: "Formatted at 2026-09-28 21:54:16 on dist-test-slave-k6lz"
server_key: "479965645448451e3eda4b9a25a66747"
server_key_iv: "e41231b4955068d0f39e715664a34437"
server_key_version: "encryptionkey@0"
I20260928 21:54:16.376222 6155 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:54:16.376997 6170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:16.377179 6155 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:54:16.377269 6155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-2/wal
uuid: "b1c3327754bc4473863b49f23c319665"
format_stamp: "Formatted at 2026-09-28 21:54:16 on dist-test-slave-k6lz"
server_key: "479965645448451e3eda4b9a25a66747"
server_key_iv: "e41231b4955068d0f39e715664a34437"
server_key_version: "encryptionkey@0"
I20260928 21:54:16.377329 6155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:16.386071 6155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:54:16.388384 6155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:54:16.388527 6155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:54:16.388751 6155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:54:16.389047 6155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:54:16.389079 6155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:16.389109 6155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:54:16.389124 6155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:16.394559 6155 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.67:41753
I20260928 21:54:16.394623 6283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.67:41753 every 8 connection(s)
I20260928 21:54:16.394910 6155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
Sep 28 21:54:16 dist-test-slave-k6lz krb5kdc[5780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632456, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.67@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:54:16.403118 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 6155
I20260928 21:54:16.403234 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632440055392-2777-0/minicluster-data/ts-2/wal/instance
I20260928 21:54:16.403508 2777 external_mini_cluster.cc:1428] Setting key 6db34f4e7e626f3414f061b00f8c4d6d
I20260928 21:54:16.404938 5882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:16.395589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.67:46731 (local address 127.2.182.126:33005)
0928 21:54:16.395637 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:16.395639 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:16.395837 (+ 198us) server_negotiation.cc:407] Connection header received
0928 21:54:16.396347 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:16.396349 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:16.396387 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:16.396437 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:16.397420 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:16.397969 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:16.398952 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:16.399158 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:16.399212 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:16.401544 (+ 2332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:16.401545 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:16.401546 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:16.401553 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:16.403054 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:16.403607 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:16.403609 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:16.403609 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:16.403773 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:16.404093 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:16.404094 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:16.404095 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:16.404335 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:16.404403 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:16.404785 (+ 382us) server_negotiation.cc:299] Negotiation successful
0928 21:54:16.404871 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 21:54:16.405161 6286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:16.395577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:33005 (local address 127.2.182.67:46731)
0928 21:54:16.395714 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:16.395752 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:16.396250 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:16.396508 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:16.396512 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:16.397131 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:16.397327 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:16.397332 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:16.398054 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:16.398055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:16.398870 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:16.398872 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:16.399227 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:16.399228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:16.399377 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:16.399949 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:16.399958 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:16.401440 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:16.403149 (+ 1709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:16.403157 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:16.403159 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:16.403507 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:16.403870 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:16.403871 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:16.403871 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:16.404021 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:16.404424 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:16.404430 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:16.404607 (+ 177us) client_negotiation.cc:769] Sending connection context
0928 21:54:16.404738 (+ 131us) client_negotiation.cc:240] Negotiation successful
0928 21:54:16.404924 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":68,"threads_started":1}
I20260928 21:54:16.405668 6284 heartbeater.cc:347] Connected to a master server at 127.2.182.126:33005
I20260928 21:54:16.405764 6284 heartbeater.cc:464] Registering TS with master...
I20260928 21:54:16.406029 6284 heartbeater.cc:511] Master 127.2.182.126:33005 requested a full tablet report, sending...
I20260928 21:54:16.406466 5825 ts_manager.cc:194] Registered new tserver with Master: b1c3327754bc4473863b49f23c319665 (127.2.182.67:41753)
I20260928 21:54:16.406910 5825 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.67@KRBTEST.COM'} at 127.2.182.67:46731
I20260928 21:54:16.416062 2777 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 21:54:16 dist-test-slave-k6lz krb5kdc[5780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632456, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 28 21:54:16 dist-test-slave-k6lz krb5kdc[5780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632456, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:54:16.440297 5882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:16.432479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53696 (local address 127.2.182.126:33005)
0928 21:54:16.432538 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:16.432540 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:16.432657 (+ 117us) server_negotiation.cc:407] Connection header received
0928 21:54:16.432692 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:16.432693 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:16.432704 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:16.432861 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:16.433534 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:16.434093 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:16.434943 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:16.435143 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:16.435210 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:16.437682 (+ 2472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:16.437684 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:16.437684 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:16.437692 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:16.438987 (+ 1295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:16.439417 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:16.439418 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:16.439419 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:16.439478 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:16.439788 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:16.439789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:16.439789 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:16.439961 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:16.440042 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:16.440151 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 21:54:16.440229 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260928 21:54:16.440531 5825 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:53696
I20260928 21:54:16.441007 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 5887
I20260928 21:54:16.445147 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 6021
I20260928 21:54:16.448840 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 6155
I20260928 21:54:16.452806 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 5793
2026-09-28T21:54:16Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1772 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:54:16 dist-test-slave-k6lz krb5kdc[6296](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:54:16 dist-test-slave-k6lz krb5kdc[6296](info): set up 2 sockets
Sep 28 21:54:16 dist-test-slave-k6lz krb5kdc[6296](info): commencing operation
krb5kdc: starting...
W20260928 21:54:17.419147 2777 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Sep 28 21:54:17 dist-test-slave-k6lz krb5kdc[6296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:54:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:54:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab.
Entry for principal kudu/127.2.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab.
I20260928 21:54:17.538594 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:36465
--webserver_interface=127.2.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab
--principal=kudu/127.2.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:32899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.126:36465
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:54:17.618309 6309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:54:17.618523 6309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:54:17.618553 6309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:54:17.619779 6309 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:54:17.619823 6309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:54:17.619840 6309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:54:17.619856 6309 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:54:17.619871 6309 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:54:17.621451 6309 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:32899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.126:36465
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab
--principal=kudu/127.2.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:36465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:54:17.621712 6309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:54:17.621944 6309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:54:17.626718 6315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:17.626729 6317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:17.626987 6309 server_base.cc:1034] running on GCE node
W20260928 21:54:17.626716 6314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:17.627282 6309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:17.627615 6309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:17.628764 6309 hybrid_clock.cc:648] HybridClock initialized: now 1790632457628743 us; error 38 us; skew 500 ppm
Sep 28 21:54:17 dist-test-slave-k6lz krb5kdc[6296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632457, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:54:17.631354 6309 init.cc:377] Logged in from keytab as kudu/127.2.182.126@KRBTEST.COM (short username kudu)
I20260928 21:54:17.631799 6309 webserver.cc:460] Webserver started at http://127.2.182.126:39995/ using document root <none> and password file <none>
I20260928 21:54:17.632021 6309 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:17.632066 6309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:17.632222 6309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:54:17.633268 6309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/minicluster-data/master-0/data/instance:
uuid: "22b9640a3aa34c47929db28f1bd6e01a"
format_stamp: "Formatted at 2026-09-28 21:54:17 on dist-test-slave-k6lz"
server_key: "93ed754f52317519d8d51c8b0db173b7"
server_key_iv: "4c266212bc3d69dd579d9a04e52d3b8f"
server_key_version: "encryptionkey@0"
I20260928 21:54:17.633584 6309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/minicluster-data/master-0/wal/instance:
uuid: "22b9640a3aa34c47929db28f1bd6e01a"
format_stamp: "Formatted at 2026-09-28 21:54:17 on dist-test-slave-k6lz"
server_key: "93ed754f52317519d8d51c8b0db173b7"
server_key_iv: "4c266212bc3d69dd579d9a04e52d3b8f"
server_key_version: "encryptionkey@0"
I20260928 21:54:17.635429 6309 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:54:17.636330 6324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:17.636554 6309 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:54:17.636652 6309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/minicluster-data/master-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/minicluster-data/master-0/wal
uuid: "22b9640a3aa34c47929db28f1bd6e01a"
format_stamp: "Formatted at 2026-09-28 21:54:17 on dist-test-slave-k6lz"
server_key: "93ed754f52317519d8d51c8b0db173b7"
server_key_iv: "4c266212bc3d69dd579d9a04e52d3b8f"
server_key_version: "encryptionkey@0"
I20260928 21:54:17.636710 6309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:17.647997 6309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:54:17.650120 6309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:54:17.650229 6309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:54:17.653882 6309 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.126:36465
I20260928 21:54:17.653941 6376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.126:36465 every 8 connection(s)
I20260928 21:54:17.654383 6309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
I20260928 21:54:17.655054 6377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:54:17.657490 6377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22b9640a3aa34c47929db28f1bd6e01a: Bootstrap starting.
I20260928 21:54:17.658073 6377 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22b9640a3aa34c47929db28f1bd6e01a: Neither blocks nor log segments found. Creating new log.
I20260928 21:54:17.658352 6377 log.cc:824] T 00000000000000000000000000000000 P 22b9640a3aa34c47929db28f1bd6e01a: Log is configured to *not* fsync() on all Append() calls
I20260928 21:54:17.659036 6377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22b9640a3aa34c47929db28f1bd6e01a: No bootstrap required, opened a new log
I20260928 21:54:17.660765 6377 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22b9640a3aa34c47929db28f1bd6e01a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b9640a3aa34c47929db28f1bd6e01a" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 36465 } }
I20260928 21:54:17.660909 6377 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22b9640a3aa34c47929db28f1bd6e01a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:54:17.660930 6377 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22b9640a3aa34c47929db28f1bd6e01a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22b9640a3aa34c47929db28f1bd6e01a, State: Initialized, Role: FOLLOWER
I20260928 21:54:17.661068 6377 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22b9640a3aa34c47929db28f1bd6e01a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b9640a3aa34c47929db28f1bd6e01a" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 36465 } }
I20260928 21:54:17.661144 6377 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22b9640a3aa34c47929db28f1bd6e01a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:54:17.661166 6377 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22b9640a3aa34c47929db28f1bd6e01a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:54:17.661197 6377 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22b9640a3aa34c47929db28f1bd6e01a [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:54:17.662079 6377 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22b9640a3aa34c47929db28f1bd6e01a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b9640a3aa34c47929db28f1bd6e01a" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 36465 } }
I20260928 21:54:17.662186 6377 leader_election.cc:302] T 00000000000000000000000000000000 P 22b9640a3aa34c47929db28f1bd6e01a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22b9640a3aa34c47929db28f1bd6e01a; no voters:
I20260928 21:54:17.662351 6377 leader_election.cc:288] T 00000000000000000000000000000000 P 22b9640a3aa34c47929db28f1bd6e01a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:54:17.662393 6380 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22b9640a3aa34c47929db28f1bd6e01a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:54:17.662554 6380 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22b9640a3aa34c47929db28f1bd6e01a [term 1 LEADER]: Becoming Leader. State: Replica: 22b9640a3aa34c47929db28f1bd6e01a, State: Running, Role: LEADER
I20260928 21:54:17.662622 6377 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22b9640a3aa34c47929db28f1bd6e01a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:54:17.662649 6380 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22b9640a3aa34c47929db28f1bd6e01a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b9640a3aa34c47929db28f1bd6e01a" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 36465 } }
I20260928 21:54:17.663089 6381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22b9640a3aa34c47929db28f1bd6e01a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "22b9640a3aa34c47929db28f1bd6e01a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b9640a3aa34c47929db28f1bd6e01a" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 36465 } } }
I20260928 21:54:17.663183 6381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22b9640a3aa34c47929db28f1bd6e01a [sys.catalog]: This master's current role is: LEADER
I20260928 21:54:17.663105 6382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22b9640a3aa34c47929db28f1bd6e01a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22b9640a3aa34c47929db28f1bd6e01a. Latest consensus state: current_term: 1 leader_uuid: "22b9640a3aa34c47929db28f1bd6e01a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b9640a3aa34c47929db28f1bd6e01a" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 36465 } } }
I20260928 21:54:17.663558 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 6309
I20260928 21:54:17.663654 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/minicluster-data/master-0/wal/instance
I20260928 21:54:17.663831 2777 external_mini_cluster.cc:1428] Setting key b9c75f65781b5f33f2ff36a1279b599d
I20260928 21:54:17.663319 6382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22b9640a3aa34c47929db28f1bd6e01a [sys.catalog]: This master's current role is: LEADER
I20260928 21:54:17.663699 6389 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:54:17.664682 6389 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:54:17.666208 6389 catalog_manager.cc:1347] Generated new cluster ID: 556c7eb2acda48fe94047f66b3f41627
I20260928 21:54:17.666261 6389 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 21:54:17 dist-test-slave-k6lz krb5kdc[6296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:54:17.673954 6399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:17.664455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49030 (local address 127.2.182.126:36465)
0928 21:54:17.665360 (+ 905us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:17.665370 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:17.665403 (+ 33us) server_negotiation.cc:407] Connection header received
0928 21:54:17.665523 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:17.665530 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:17.665584 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:17.665894 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:17.666574 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:17.667636 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:17.668359 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:17.668776 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:17.668875 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:17.670954 (+ 2079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:17.670959 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:17.670959 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:17.670972 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:17.672524 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:17.672834 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:17.672837 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:17.672838 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:17.672930 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:17.673096 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:17.673096 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:17.673097 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:17.673348 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:17.673496 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:17.673592 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 21:54:17.673744 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":860,"thread_start_us":78,"threads_started":1}
I20260928 21:54:17.709365 6389 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:54:17.709910 6389 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:54:17.716969 6389 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 22b9640a3aa34c47929db28f1bd6e01a: Generated new TSK 0
I20260928 21:54:17.717178 6389 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.182.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab.
Entry for principal kudu/127.2.182.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab.
I20260928 21:54:17.755268 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.65:0
--local_ip_for_outbound_sockets=127.2.182.65
--webserver_interface=127.2.182.65
--webserver_port=0
--tserver_master_addrs=127.2.182.126:36465
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab
--principal=kudu/127.2.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:32899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:54:17.846176 6403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:54:17.846400 6403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:54:17.846426 6403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:54:17.847675 6403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:54:17.847735 6403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:54:17.847772 6403 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.65
I20260928 21:54:17.849364 6403 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:32899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab
--principal=kudu/127.2.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.182.65
--webserver_port=0
--tserver_master_addrs=127.2.182.126:36465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6403
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.65
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:54:17.849606 6403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:54:17.849841 6403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:54:17.855549 6403 server_base.cc:1034] running on GCE node
W20260928 21:54:17.855535 6409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:17.855487 6411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:17.855490 6408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:17.855986 6403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:17.856281 6403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:17.857447 6403 hybrid_clock.cc:648] HybridClock initialized: now 1790632457857413 us; error 46 us; skew 500 ppm
Sep 28 21:54:17 dist-test-slave-k6lz krb5kdc[6296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632457, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:54:17.866242 6403 init.cc:377] Logged in from keytab as kudu/127.2.182.65@KRBTEST.COM (short username kudu)
I20260928 21:54:17.866729 6403 webserver.cc:460] Webserver started at http://127.2.182.65:34113/ using document root <none> and password file <none>
I20260928 21:54:17.866954 6403 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:17.867005 6403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:17.867115 6403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:54:17.868513 6403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/minicluster-data/ts-0/data/instance:
uuid: "8742870e97a54f3aa7b2c8221264d0ff"
format_stamp: "Formatted at 2026-09-28 21:54:17 on dist-test-slave-k6lz"
server_key: "e85865bdb0f60e86a6f6cde5cfd8ff8b"
server_key_iv: "a4aaf41e4f7f79e2dc734017e112be92"
server_key_version: "encryptionkey@0"
I20260928 21:54:17.868844 6403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/minicluster-data/ts-0/wal/instance:
uuid: "8742870e97a54f3aa7b2c8221264d0ff"
format_stamp: "Formatted at 2026-09-28 21:54:17 on dist-test-slave-k6lz"
server_key: "e85865bdb0f60e86a6f6cde5cfd8ff8b"
server_key_iv: "a4aaf41e4f7f79e2dc734017e112be92"
server_key_version: "encryptionkey@0"
I20260928 21:54:17.870265 6403 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:54:17.871302 6418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:17.871560 6403 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:54:17.871644 6403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/minicluster-data/ts-0/wal
uuid: "8742870e97a54f3aa7b2c8221264d0ff"
format_stamp: "Formatted at 2026-09-28 21:54:17 on dist-test-slave-k6lz"
server_key: "e85865bdb0f60e86a6f6cde5cfd8ff8b"
server_key_iv: "a4aaf41e4f7f79e2dc734017e112be92"
server_key_version: "encryptionkey@0"
I20260928 21:54:17.871707 6403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:17.879621 6403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:54:17.881731 6403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:54:17.881860 6403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:54:17.882064 6403 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:54:17.882331 6403 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:54:17.882361 6403 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:17.882383 6403 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:54:17.882395 6403 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:17.888268 6403 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.65:46245
I20260928 21:54:17.888274 6531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.65:46245 every 8 connection(s)
I20260928 21:54:17.888639 6403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
I20260928 21:54:17.890949 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 6403
I20260928 21:54:17.891058 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/minicluster-data/ts-0/wal/instance
I20260928 21:54:17.891294 2777 external_mini_cluster.cc:1428] Setting key c2724f979adc24ac8cdce7cfe5f2d5a1
Sep 28 21:54:17 dist-test-slave-k6lz krb5kdc[6296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632457, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.65@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:54:17.898895 6399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:17.889368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.65:51853 (local address 127.2.182.126:36465)
0928 21:54:17.889417 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:17.889420 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:17.889606 (+ 186us) server_negotiation.cc:407] Connection header received
0928 21:54:17.890620 (+ 1014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:17.890621 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:17.890638 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:17.890690 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:17.891586 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:17.892147 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:17.893319 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:17.893650 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:17.893733 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:17.896278 (+ 2545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:17.896280 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:17.896281 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:17.896292 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:17.897568 (+ 1276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:17.897970 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:17.897971 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:17.897972 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:17.898027 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:17.898248 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:17.898249 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:17.898249 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:17.898365 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:17.898425 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:17.898697 (+ 272us) server_negotiation.cc:299] Negotiation successful
0928 21:54:17.898799 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
WARNING: no policy specified for kudu/127.2.182.66@KRBTEST.COM; defaulting to no policy
I20260928 21:54:17.899085 6534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:17.889310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:36465 (local address 127.2.182.65:51853)
0928 21:54:17.889486 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:17.889525 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:17.890509 (+ 984us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:17.890792 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:17.890796 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:17.891163 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:17.891486 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:17.891492 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:17.892259 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:17.892260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:17.893230 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:17.893233 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:17.893833 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:17.893834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:17.893953 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:17.894483 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:17.894493 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:17.896176 (+ 1683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:17.897650 (+ 1474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:17.897658 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:17.897661 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:17.897901 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:17.898080 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:17.898081 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:17.898081 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:17.898189 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:17.898420 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:17.898427 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:17.898575 (+ 148us) client_negotiation.cc:769] Sending connection context
0928 21:54:17.898667 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 21:54:17.898857 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":79,"threads_started":1}
I20260928 21:54:17.900242 6532 heartbeater.cc:347] Connected to a master server at 127.2.182.126:36465
I20260928 21:54:17.900337 6532 heartbeater.cc:464] Registering TS with master...
I20260928 21:54:17.900511 6532 heartbeater.cc:511] Master 127.2.182.126:36465 requested a full tablet report, sending...
I20260928 21:54:17.901239 6341 ts_manager.cc:194] Registered new tserver with Master: 8742870e97a54f3aa7b2c8221264d0ff (127.2.182.65:46245)
I20260928 21:54:17.901927 6341 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.65@KRBTEST.COM'} at 127.2.182.65:51853
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab.
Entry for principal kudu/127.2.182.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab.
I20260928 21:54:17.919076 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.66:0
--local_ip_for_outbound_sockets=127.2.182.66
--webserver_interface=127.2.182.66
--webserver_port=0
--tserver_master_addrs=127.2.182.126:36465
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab
--principal=kudu/127.2.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:32899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:54:18.000219 6537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:54:18.000440 6537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:54:18.000473 6537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:54:18.001842 6537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:54:18.001916 6537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:54:18.001959 6537 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.66
I20260928 21:54:18.003685 6537 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:32899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab
--principal=kudu/127.2.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.182.66
--webserver_port=0
--tserver_master_addrs=127.2.182.126:36465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6537
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.66
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:54:18.003952 6537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:54:18.004171 6537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:54:18.009259 6542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:18.009315 6543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:18.009256 6545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:18.009656 6537 server_base.cc:1034] running on GCE node
I20260928 21:54:18.009863 6537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:18.010084 6537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:18.011266 6537 hybrid_clock.cc:648] HybridClock initialized: now 1790632458011251 us; error 26 us; skew 500 ppm
Sep 28 21:54:18 dist-test-slave-k6lz krb5kdc[6296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632458, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:54:18.014050 6537 init.cc:377] Logged in from keytab as kudu/127.2.182.66@KRBTEST.COM (short username kudu)
I20260928 21:54:18.014508 6537 webserver.cc:460] Webserver started at http://127.2.182.66:42829/ using document root <none> and password file <none>
I20260928 21:54:18.014708 6537 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:18.014750 6537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:18.014909 6537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:54:18.016319 6537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/minicluster-data/ts-1/data/instance:
uuid: "ce80ddd5dc2e48a595fdf9f03f02b79f"
format_stamp: "Formatted at 2026-09-28 21:54:18 on dist-test-slave-k6lz"
server_key: "9437577c5c3d477c39b74385b94d8a50"
server_key_iv: "8886beab0d25f2d2f32cec3472ac22b2"
server_key_version: "encryptionkey@0"
I20260928 21:54:18.016708 6537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/minicluster-data/ts-1/wal/instance:
uuid: "ce80ddd5dc2e48a595fdf9f03f02b79f"
format_stamp: "Formatted at 2026-09-28 21:54:18 on dist-test-slave-k6lz"
server_key: "9437577c5c3d477c39b74385b94d8a50"
server_key_iv: "8886beab0d25f2d2f32cec3472ac22b2"
server_key_version: "encryptionkey@0"
I20260928 21:54:18.018415 6537 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260928 21:54:18.019310 6552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:18.019536 6537 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:54:18.019652 6537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/minicluster-data/ts-1/wal
uuid: "ce80ddd5dc2e48a595fdf9f03f02b79f"
format_stamp: "Formatted at 2026-09-28 21:54:18 on dist-test-slave-k6lz"
server_key: "9437577c5c3d477c39b74385b94d8a50"
server_key_iv: "8886beab0d25f2d2f32cec3472ac22b2"
server_key_version: "encryptionkey@0"
I20260928 21:54:18.019747 6537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:18.028343 6537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:54:18.030551 6537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:54:18.030709 6537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:54:18.031107 6537 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:54:18.031599 6537 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:54:18.031641 6537 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:18.031672 6537 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:54:18.031698 6537 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:18.037160 6537 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.66:33101
I20260928 21:54:18.037207 6665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.66:33101 every 8 connection(s)
I20260928 21:54:18.037509 6537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
Sep 28 21:54:18 dist-test-slave-k6lz krb5kdc[6296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632458, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.66@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:54:18.044178 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 6537
I20260928 21:54:18.044271 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/minicluster-data/ts-1/wal/instance
I20260928 21:54:18.044449 2777 external_mini_cluster.cc:1428] Setting key be1d7d5676176d56139d69af9367a07a
I20260928 21:54:18.047612 6399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:18.038224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.66:56061 (local address 127.2.182.126:36465)
0928 21:54:18.038292 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:18.038294 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:18.038576 (+ 282us) server_negotiation.cc:407] Connection header received
0928 21:54:18.039601 (+ 1025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:18.039603 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:18.039616 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:18.039669 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:18.040292 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:18.040922 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:18.041890 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:18.042078 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:18.042134 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:18.044512 (+ 2378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:18.044514 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:18.044514 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:18.044523 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:18.046088 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:18.046526 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:18.046527 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:18.046527 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:18.046594 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:18.046879 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:18.046880 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:18.046880 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:18.046993 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:18.047042 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:18.047443 (+ 401us) server_negotiation.cc:299] Negotiation successful
0928 21:54:18.047540 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 21:54:18.047824 6668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:18.038170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:36465 (local address 127.2.182.66:56061)
0928 21:54:18.038438 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:18.038478 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:18.039479 (+ 1001us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:18.039736 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:18.039739 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:18.040020 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:18.040225 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:18.040229 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:18.041008 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:18.041009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:18.041801 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:18.041803 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:18.042156 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:18.042157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:18.042264 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:18.042770 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:18.042779 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:18.044419 (+ 1640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:18.046182 (+ 1763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:18.046191 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:18.046194 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:18.046441 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:18.046657 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:18.046658 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:18.046658 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:18.046817 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:18.047047 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:18.047053 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:18.047225 (+ 172us) client_negotiation.cc:769] Sending connection context
0928 21:54:18.047416 (+ 191us) client_negotiation.cc:240] Negotiation successful
0928 21:54:18.047600 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":145,"threads_started":1}
I20260928 21:54:18.048360 6666 heartbeater.cc:347] Connected to a master server at 127.2.182.126:36465
I20260928 21:54:18.048458 6666 heartbeater.cc:464] Registering TS with master...
I20260928 21:54:18.048638 6666 heartbeater.cc:511] Master 127.2.182.126:36465 requested a full tablet report, sending...
I20260928 21:54:18.049021 6341 ts_manager.cc:194] Registered new tserver with Master: ce80ddd5dc2e48a595fdf9f03f02b79f (127.2.182.66:33101)
I20260928 21:54:18.049397 6341 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.66@KRBTEST.COM'} at 127.2.182.66:56061
WARNING: no policy specified for kudu/127.2.182.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab.
Entry for principal kudu/127.2.182.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab.
I20260928 21:54:18.070672 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.67:0
--local_ip_for_outbound_sockets=127.2.182.67
--webserver_interface=127.2.182.67
--webserver_port=0
--tserver_master_addrs=127.2.182.126:36465
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab
--principal=kudu/127.2.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:32899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:54:18.156566 6671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:54:18.156792 6671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:54:18.156826 6671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:54:18.158073 6671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:54:18.158123 6671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:54:18.158169 6671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.67
I20260928 21:54:18.159812 6671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:32899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab
--principal=kudu/127.2.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.182.67
--webserver_port=0
--tserver_master_addrs=127.2.182.126:36465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.67
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:54:18.160066 6671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:54:18.160286 6671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:54:18.165582 6676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:18.165967 6679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:18.166101 6671 server_base.cc:1034] running on GCE node
W20260928 21:54:18.166262 6677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:18.166507 6671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:18.166728 6671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:18.167896 6671 hybrid_clock.cc:648] HybridClock initialized: now 1790632458167879 us; error 29 us; skew 500 ppm
Sep 28 21:54:18 dist-test-slave-k6lz krb5kdc[6296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632458, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:54:18.170475 6671 init.cc:377] Logged in from keytab as kudu/127.2.182.67@KRBTEST.COM (short username kudu)
I20260928 21:54:18.171063 6671 webserver.cc:460] Webserver started at http://127.2.182.67:40975/ using document root <none> and password file <none>
I20260928 21:54:18.171279 6671 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:18.171358 6671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:18.171487 6671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:54:18.172753 6671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/minicluster-data/ts-2/data/instance:
uuid: "459184644975495895952ea52a4e2b33"
format_stamp: "Formatted at 2026-09-28 21:54:18 on dist-test-slave-k6lz"
server_key: "fd134f86be773ce15ccbc74200d04f85"
server_key_iv: "7b3b0eb0018e2ae572b640c418e474a2"
server_key_version: "encryptionkey@0"
I20260928 21:54:18.173115 6671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/minicluster-data/ts-2/wal/instance:
uuid: "459184644975495895952ea52a4e2b33"
format_stamp: "Formatted at 2026-09-28 21:54:18 on dist-test-slave-k6lz"
server_key: "fd134f86be773ce15ccbc74200d04f85"
server_key_iv: "7b3b0eb0018e2ae572b640c418e474a2"
server_key_version: "encryptionkey@0"
I20260928 21:54:18.174562 6671 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 21:54:18.175469 6686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:18.175627 6671 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:54:18.175709 6671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/minicluster-data/ts-2/wal
uuid: "459184644975495895952ea52a4e2b33"
format_stamp: "Formatted at 2026-09-28 21:54:18 on dist-test-slave-k6lz"
server_key: "fd134f86be773ce15ccbc74200d04f85"
server_key_iv: "7b3b0eb0018e2ae572b640c418e474a2"
server_key_version: "encryptionkey@0"
I20260928 21:54:18.175769 6671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:18.188957 6671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:54:18.191228 6671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:54:18.191402 6671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:54:18.191609 6671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:54:18.191890 6671 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:54:18.191922 6671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:18.191943 6671 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:54:18.191954 6671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:18.197945 6671 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.67:46167
I20260928 21:54:18.198022 6799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.67:46167 every 8 connection(s)
I20260928 21:54:18.198391 6671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
Sep 28 21:54:18 dist-test-slave-k6lz krb5kdc[6296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632458, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.67@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:54:18.206091 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 6671
I20260928 21:54:18.206190 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632440055392-2777-0/minicluster-data/ts-2/wal/instance
I20260928 21:54:18.206360 2777 external_mini_cluster.cc:1428] Setting key d73965ac945d16cb76e1ed682afa65af
I20260928 21:54:18.208536 6399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:18.199269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.67:59713 (local address 127.2.182.126:36465)
0928 21:54:18.199391 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:18.199394 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:18.199562 (+ 168us) server_negotiation.cc:407] Connection header received
0928 21:54:18.200605 (+ 1043us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:18.200607 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:18.200619 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:18.200673 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:18.201316 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:18.201812 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:18.202816 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:18.203006 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:18.203073 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:18.205335 (+ 2262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:18.205336 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:18.205337 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:18.205344 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:18.207006 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:18.207522 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:18.207523 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:18.207524 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:18.207583 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:18.207894 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:18.207895 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:18.207896 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:18.208027 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:18.208081 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:18.208363 (+ 282us) server_negotiation.cc:299] Negotiation successful
0928 21:54:18.208443 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 21:54:18.208700 6802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:18.199215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:36465 (local address 127.2.182.67:59713)
0928 21:54:18.199437 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:18.199479 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:18.200492 (+ 1013us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:18.200747 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:18.200750 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:18.201027 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:18.201226 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:18.201231 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:18.201903 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:18.201904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:18.202712 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:18.202715 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:18.203082 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:18.203082 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:18.203202 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:18.203748 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:18.203757 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:18.205240 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:18.207114 (+ 1874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:18.207121 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:18.207123 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:18.207460 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:18.207636 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:18.207637 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:18.207637 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:18.207761 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:18.208084 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:18.208091 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:18.208229 (+ 138us) client_negotiation.cc:769] Sending connection context
0928 21:54:18.208318 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 21:54:18.208484 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":117,"threads_started":1}
I20260928 21:54:18.209348 6800 heartbeater.cc:347] Connected to a master server at 127.2.182.126:36465
I20260928 21:54:18.209437 6800 heartbeater.cc:464] Registering TS with master...
I20260928 21:54:18.209592 6800 heartbeater.cc:511] Master 127.2.182.126:36465 requested a full tablet report, sending...
I20260928 21:54:18.209967 6341 ts_manager.cc:194] Registered new tserver with Master: 459184644975495895952ea52a4e2b33 (127.2.182.67:46167)
I20260928 21:54:18.210314 6341 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.67@KRBTEST.COM'} at 127.2.182.67:59713
I20260928 21:54:18.218725 2777 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 21:54:18 dist-test-slave-k6lz krb5kdc[6296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 21:54:18 dist-test-slave-k6lz krb5kdc[6296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.182.65@KRBTEST.COM
I20260928 21:54:18.243194 6807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:18.234253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47504 (local address 127.2.182.65:46245)
0928 21:54:18.234412 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:18.234417 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:18.234427 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:54:18.234453 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:18.234456 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:18.234475 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:18.234542 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:18.235142 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:18.236022 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:18.236910 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:18.237286 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:18.237353 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:18.240113 (+ 2760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:18.240124 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:18.240124 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:18.240133 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:18.242023 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:18.242402 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:18.242406 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:18.242406 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:18.242475 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:18.242642 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:18.242643 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:18.242643 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:18.242784 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:18.242966 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:18.243043 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 21:54:18.243113 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":75,"threads_started":1}
Sep 28 21:54:18 dist-test-slave-k6lz krb5kdc[6296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:54:18.252399 6399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:18.244525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49034 (local address 127.2.182.126:36465)
0928 21:54:18.244581 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:18.244583 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:18.244595 (+ 12us) server_negotiation.cc:407] Connection header received
0928 21:54:18.244646 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:18.244647 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:18.244655 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:18.244698 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:18.245295 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:18.245910 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:18.246772 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:18.247056 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:18.247211 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:18.249563 (+ 2352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:18.249565 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:18.249565 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:18.249574 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:18.250912 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:18.251367 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:18.251369 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:18.251369 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:18.251431 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:18.251755 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:18.251756 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:18.251756 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:18.251934 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:18.252018 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:18.252218 (+ 200us) server_negotiation.cc:299] Negotiation successful
0928 21:54:18.252290 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
W20260928 21:54:18.252714 6341 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49034
I20260928 21:54:18.253428 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 6403
I20260928 21:54:18.257763 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 6537
I20260928 21:54:18.261300 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 6671
I20260928 21:54:18.265347 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 6309
2026-09-28T21:54:18Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1813 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:54:18 dist-test-slave-k6lz krb5kdc[6812](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:54:18 dist-test-slave-k6lz krb5kdc[6812](info): set up 2 sockets
Sep 28 21:54:18 dist-test-slave-k6lz krb5kdc[6812](info): commencing operation
krb5kdc: starting...
W20260928 21:54:19.235312 2777 mini_kdc.cc:121] Time spent starting KDC: real 0.951s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Sep 28 21:54:19 dist-test-slave-k6lz krb5kdc[6812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632459, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:54:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:54:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab.
Entry for principal kudu/127.2.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab.
I20260928 21:54:19.368626 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:34355
--webserver_interface=127.2.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab
--principal=kudu/127.2.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:41027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.126:34355
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:54:19.456209 6825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:54:19.456450 6825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:54:19.456486 6825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:54:19.457877 6825 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:54:19.457952 6825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:54:19.457973 6825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:54:19.457993 6825 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:54:19.458009 6825 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:54:19.459990 6825 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:41027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.126:34355
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab
--principal=kudu/127.2.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:34355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6825
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:54:19.460279 6825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:54:19.460493 6825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:54:19.465648 6825 server_base.cc:1034] running on GCE node
W20260928 21:54:19.465613 6831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:19.465619 6830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:19.465623 6833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:19.466120 6825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:19.466398 6825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:19.467536 6825 hybrid_clock.cc:648] HybridClock initialized: now 1790632459467523 us; error 34 us; skew 500 ppm
Sep 28 21:54:19 dist-test-slave-k6lz krb5kdc[6812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632459, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:54:19.470088 6825 init.cc:377] Logged in from keytab as kudu/127.2.182.126@KRBTEST.COM (short username kudu)
I20260928 21:54:19.470475 6825 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:19.470530 6825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:19.470656 6825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:54:19.471783 6825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/minicluster-data/master-0/data/instance:
uuid: "df90d7d076d24d03beddac0168792541"
format_stamp: "Formatted at 2026-09-28 21:54:19 on dist-test-slave-k6lz"
server_key: "cb9043825b530d1102d69c6b0e957cb9"
server_key_iv: "61b862f0199bb382b0e32812fa262fc2"
server_key_version: "encryptionkey@0"
I20260928 21:54:19.472157 6825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/minicluster-data/master-0/wal/instance:
uuid: "df90d7d076d24d03beddac0168792541"
format_stamp: "Formatted at 2026-09-28 21:54:19 on dist-test-slave-k6lz"
server_key: "cb9043825b530d1102d69c6b0e957cb9"
server_key_iv: "61b862f0199bb382b0e32812fa262fc2"
server_key_version: "encryptionkey@0"
I20260928 21:54:19.473516 6825 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:54:19.474462 6839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:19.474771 6825 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:54:19.474870 6825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/minicluster-data/master-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/minicluster-data/master-0/wal
uuid: "df90d7d076d24d03beddac0168792541"
format_stamp: "Formatted at 2026-09-28 21:54:19 on dist-test-slave-k6lz"
server_key: "cb9043825b530d1102d69c6b0e957cb9"
server_key_iv: "61b862f0199bb382b0e32812fa262fc2"
server_key_version: "encryptionkey@0"
I20260928 21:54:19.474937 6825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:19.489130 6825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:54:19.491015 6825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:54:19.491154 6825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:54:19.495041 6825 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.126:34355
I20260928 21:54:19.495077 6891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.126:34355 every 8 connection(s)
I20260928 21:54:19.495385 6825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
I20260928 21:54:19.496390 6892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:54:19.498749 6892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df90d7d076d24d03beddac0168792541: Bootstrap starting.
I20260928 21:54:19.499398 6892 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df90d7d076d24d03beddac0168792541: Neither blocks nor log segments found. Creating new log.
I20260928 21:54:19.499670 6892 log.cc:824] T 00000000000000000000000000000000 P df90d7d076d24d03beddac0168792541: Log is configured to *not* fsync() on all Append() calls
I20260928 21:54:19.500500 6892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df90d7d076d24d03beddac0168792541: No bootstrap required, opened a new log
I20260928 21:54:19.502113 6892 raft_consensus.cc:348] T 00000000000000000000000000000000 P df90d7d076d24d03beddac0168792541 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df90d7d076d24d03beddac0168792541" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 34355 } }
I20260928 21:54:19.502269 6892 raft_consensus.cc:374] T 00000000000000000000000000000000 P df90d7d076d24d03beddac0168792541 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:54:19.502291 6892 raft_consensus.cc:729] T 00000000000000000000000000000000 P df90d7d076d24d03beddac0168792541 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df90d7d076d24d03beddac0168792541, State: Initialized, Role: FOLLOWER
I20260928 21:54:19.502425 6892 consensus_queue.cc:260] T 00000000000000000000000000000000 P df90d7d076d24d03beddac0168792541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df90d7d076d24d03beddac0168792541" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 34355 } }
I20260928 21:54:19.502496 6892 raft_consensus.cc:388] T 00000000000000000000000000000000 P df90d7d076d24d03beddac0168792541 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:54:19.502530 6892 raft_consensus.cc:482] T 00000000000000000000000000000000 P df90d7d076d24d03beddac0168792541 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:54:19.502578 6892 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df90d7d076d24d03beddac0168792541 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:54:19.503460 6892 raft_consensus.cc:504] T 00000000000000000000000000000000 P df90d7d076d24d03beddac0168792541 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df90d7d076d24d03beddac0168792541" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 34355 } }
I20260928 21:54:19.503553 6892 leader_election.cc:302] T 00000000000000000000000000000000 P df90d7d076d24d03beddac0168792541 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df90d7d076d24d03beddac0168792541; no voters:
I20260928 21:54:19.503741 6892 leader_election.cc:288] T 00000000000000000000000000000000 P df90d7d076d24d03beddac0168792541 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:54:19.503788 6895 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df90d7d076d24d03beddac0168792541 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:54:19.503892 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 6825
I20260928 21:54:19.503944 6895 raft_consensus.cc:686] T 00000000000000000000000000000000 P df90d7d076d24d03beddac0168792541 [term 1 LEADER]: Becoming Leader. State: Replica: df90d7d076d24d03beddac0168792541, State: Running, Role: LEADER
I20260928 21:54:19.503986 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/minicluster-data/master-0/wal/instance
I20260928 21:54:19.504031 6892 sys_catalog.cc:564] T 00000000000000000000000000000000 P df90d7d076d24d03beddac0168792541 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:54:19.504036 6895 consensus_queue.cc:237] T 00000000000000000000000000000000 P df90d7d076d24d03beddac0168792541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df90d7d076d24d03beddac0168792541" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 34355 } }
I20260928 21:54:19.504233 2777 external_mini_cluster.cc:1428] Setting key e1ba69a87179273b28fcb64124bf5693
I20260928 21:54:19.504472 6897 sys_catalog.cc:455] T 00000000000000000000000000000000 P df90d7d076d24d03beddac0168792541 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df90d7d076d24d03beddac0168792541" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df90d7d076d24d03beddac0168792541" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 34355 } } }
I20260928 21:54:19.504570 6897 sys_catalog.cc:458] T 00000000000000000000000000000000 P df90d7d076d24d03beddac0168792541 [sys.catalog]: This master's current role is: LEADER
I20260928 21:54:19.504462 6895 sys_catalog.cc:455] T 00000000000000000000000000000000 P df90d7d076d24d03beddac0168792541 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df90d7d076d24d03beddac0168792541. Latest consensus state: current_term: 1 leader_uuid: "df90d7d076d24d03beddac0168792541" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df90d7d076d24d03beddac0168792541" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 34355 } } }
I20260928 21:54:19.504750 6895 sys_catalog.cc:458] T 00000000000000000000000000000000 P df90d7d076d24d03beddac0168792541 [sys.catalog]: This master's current role is: LEADER
I20260928 21:54:19.504859 6902 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:54:19.505522 6902 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:54:19.507508 6902 catalog_manager.cc:1347] Generated new cluster ID: 75a687d4794247509762b6239d2dfb9c
I20260928 21:54:19.507570 6902 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 21:54:19 dist-test-slave-k6lz krb5kdc[6812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632459, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:54:19.517704 6906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:19.504879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59396 (local address 127.2.182.126:34355)
0928 21:54:19.505550 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:19.505562 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:19.505597 (+ 35us) server_negotiation.cc:407] Connection header received
0928 21:54:19.505724 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:19.505732 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:19.505791 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:19.505936 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:19.507497 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:19.508963 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:19.510133 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:19.510806 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:19.510927 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:19.513914 (+ 2987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:19.513920 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:19.513921 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:19.513936 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:19.515968 (+ 2032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:19.516399 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:19.516402 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:19.516403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:19.516524 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:19.516727 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:19.516728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:19.516728 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:19.517042 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:19.517202 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:19.517319 (+ 117us) server_negotiation.cc:299] Negotiation successful
0928 21:54:19.517461 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":64,"threads_started":1}
I20260928 21:54:19.519366 6902 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:54:19.520125 6902 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:54:19.533717 6902 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df90d7d076d24d03beddac0168792541: Generated new TSK 0
I20260928 21:54:19.533931 6902 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.182.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab.
Entry for principal kudu/127.2.182.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab.
I20260928 21:54:19.563796 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.65:0
--local_ip_for_outbound_sockets=127.2.182.65
--webserver_interface=127.2.182.65
--webserver_port=0
--tserver_master_addrs=127.2.182.126:34355
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab
--principal=kudu/127.2.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:41027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:54:19.646960 6918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:54:19.647186 6918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:54:19.647213 6918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:54:19.648489 6918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:54:19.648541 6918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:54:19.648571 6918 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.65
I20260928 21:54:19.650182 6918 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:41027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab
--principal=kudu/127.2.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.182.65
--webserver_port=0
--tserver_master_addrs=127.2.182.126:34355
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6918
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.65
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:54:19.650426 6918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:54:19.650671 6918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:54:19.656015 6926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:19.656019 6924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:19.656015 6923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:19.656406 6918 server_base.cc:1034] running on GCE node
I20260928 21:54:19.656634 6918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:19.656975 6918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:19.658157 6918 hybrid_clock.cc:648] HybridClock initialized: now 1790632459658138 us; error 32 us; skew 500 ppm
Sep 28 21:54:19 dist-test-slave-k6lz krb5kdc[6812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632459, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:54:19.661070 6918 init.cc:377] Logged in from keytab as kudu/127.2.182.65@KRBTEST.COM (short username kudu)
I20260928 21:54:19.661444 6918 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:19.661499 6918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:19.661644 6918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:54:19.662726 6918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/minicluster-data/ts-0/data/instance:
uuid: "4dfc8804c61742f7b54798c0b0533d98"
format_stamp: "Formatted at 2026-09-28 21:54:19 on dist-test-slave-k6lz"
server_key: "8738e0206ef6ebc9996d6ccb8ced4be6"
server_key_iv: "645337249ea4ecb11ab89f6ba1550082"
server_key_version: "encryptionkey@0"
I20260928 21:54:19.663148 6918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/minicluster-data/ts-0/wal/instance:
uuid: "4dfc8804c61742f7b54798c0b0533d98"
format_stamp: "Formatted at 2026-09-28 21:54:19 on dist-test-slave-k6lz"
server_key: "8738e0206ef6ebc9996d6ccb8ced4be6"
server_key_iv: "645337249ea4ecb11ab89f6ba1550082"
server_key_version: "encryptionkey@0"
I20260928 21:54:19.664533 6918 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:54:19.665482 6932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:19.665688 6918 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:54:19.665848 6918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/minicluster-data/ts-0/wal
uuid: "4dfc8804c61742f7b54798c0b0533d98"
format_stamp: "Formatted at 2026-09-28 21:54:19 on dist-test-slave-k6lz"
server_key: "8738e0206ef6ebc9996d6ccb8ced4be6"
server_key_iv: "645337249ea4ecb11ab89f6ba1550082"
server_key_version: "encryptionkey@0"
I20260928 21:54:19.665916 6918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:19.703835 6918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:54:19.706255 6918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:54:19.706374 6918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:54:19.706575 6918 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:54:19.706861 6918 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:54:19.706899 6918 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:19.706923 6918 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:54:19.706933 6918 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:19.713266 6918 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.65:37043
I20260928 21:54:19.713313 7045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.65:37043 every 8 connection(s)
I20260928 21:54:19.713586 6918 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
I20260928 21:54:19.719367 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 6918
I20260928 21:54:19.719477 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/minicluster-data/ts-0/wal/instance
I20260928 21:54:19.719652 2777 external_mini_cluster.cc:1428] Setting key ad12ca0a44dcc1e3b34746e1a6c761cc
Sep 28 21:54:19 dist-test-slave-k6lz krb5kdc[6812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632459, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.65@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:54:19.723940 6906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:19.714396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.65:43611 (local address 127.2.182.126:34355)
0928 21:54:19.714488 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:19.714490 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:19.714616 (+ 126us) server_negotiation.cc:407] Connection header received
0928 21:54:19.715116 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:19.715117 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:19.715132 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:19.715213 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:19.716295 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:19.716903 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:19.718033 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:19.718255 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:19.718320 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:19.720711 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:19.720713 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:19.720713 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:19.720721 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:19.722411 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:19.722860 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:19.722861 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:19.722862 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:19.722928 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:19.723239 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:19.723240 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:19.723240 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:19.723410 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:19.723473 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:19.723735 (+ 262us) server_negotiation.cc:299] Negotiation successful
0928 21:54:19.723835 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 21:54:19.724085 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:19.714326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:34355 (local address 127.2.182.65:43611)
0928 21:54:19.714494 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:19.714536 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:19.715002 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:19.715359 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:19.715364 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:19.715642 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:19.716176 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:19.716182 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:19.717011 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:19.717013 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:19.717927 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:19.717930 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:19.718351 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:19.718352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:19.718468 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:19.719059 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:19.719067 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:19.720615 (+ 1548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:19.722524 (+ 1909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:19.722532 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:19.722534 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:19.722784 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:19.723030 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:19.723031 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:19.723031 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:19.723156 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:19.723462 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:19.723466 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:19.723598 (+ 132us) client_negotiation.cc:769] Sending connection context
0928 21:54:19.723698 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 21:54:19.723877 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":93,"threads_started":1}
I20260928 21:54:19.724603 7046 heartbeater.cc:347] Connected to a master server at 127.2.182.126:34355
I20260928 21:54:19.724731 7046 heartbeater.cc:464] Registering TS with master...
I20260928 21:54:19.724925 7046 heartbeater.cc:511] Master 127.2.182.126:34355 requested a full tablet report, sending...
I20260928 21:54:19.725553 6856 ts_manager.cc:194] Registered new tserver with Master: 4dfc8804c61742f7b54798c0b0533d98 (127.2.182.65:37043)
I20260928 21:54:19.726183 6856 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.65@KRBTEST.COM'} at 127.2.182.65:43611
WARNING: no policy specified for kudu/127.2.182.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab.
Entry for principal kudu/127.2.182.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab.
I20260928 21:54:19.748006 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.66:0
--local_ip_for_outbound_sockets=127.2.182.66
--webserver_interface=127.2.182.66
--webserver_port=0
--tserver_master_addrs=127.2.182.126:34355
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab
--principal=kudu/127.2.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:41027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:54:19.830528 7051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:54:19.830775 7051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:54:19.830803 7051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:54:19.832098 7051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:54:19.832144 7051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:54:19.832172 7051 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.66
I20260928 21:54:19.833796 7051 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:41027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab
--principal=kudu/127.2.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.182.66
--webserver_port=0
--tserver_master_addrs=127.2.182.126:34355
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7051
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.66
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:54:19.834005 7051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:54:19.834190 7051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:54:19.839668 7051 server_base.cc:1034] running on GCE node
W20260928 21:54:19.839591 7057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:19.839605 7059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:19.839591 7056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:19.840075 7051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:19.840306 7051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:19.841467 7051 hybrid_clock.cc:648] HybridClock initialized: now 1790632459841454 us; error 25 us; skew 500 ppm
Sep 28 21:54:19 dist-test-slave-k6lz krb5kdc[6812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632459, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:54:19.844496 7051 init.cc:377] Logged in from keytab as kudu/127.2.182.66@KRBTEST.COM (short username kudu)
I20260928 21:54:19.844964 7051 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:19.845026 7051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:19.845122 7051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:54:19.846187 7051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/minicluster-data/ts-1/data/instance:
uuid: "baba5f95f508446e927b82d746a50152"
format_stamp: "Formatted at 2026-09-28 21:54:19 on dist-test-slave-k6lz"
server_key: "8c76023ae17830b5c22f3382b7d5dfc0"
server_key_iv: "3d8bf1dfb5833e9c131cf8c7b06a38c0"
server_key_version: "encryptionkey@0"
I20260928 21:54:19.846508 7051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/minicluster-data/ts-1/wal/instance:
uuid: "baba5f95f508446e927b82d746a50152"
format_stamp: "Formatted at 2026-09-28 21:54:19 on dist-test-slave-k6lz"
server_key: "8c76023ae17830b5c22f3382b7d5dfc0"
server_key_iv: "3d8bf1dfb5833e9c131cf8c7b06a38c0"
server_key_version: "encryptionkey@0"
I20260928 21:54:19.847846 7051 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 21:54:19.848604 7065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:19.848760 7051 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 21:54:19.848847 7051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/minicluster-data/ts-1/wal
uuid: "baba5f95f508446e927b82d746a50152"
format_stamp: "Formatted at 2026-09-28 21:54:19 on dist-test-slave-k6lz"
server_key: "8c76023ae17830b5c22f3382b7d5dfc0"
server_key_iv: "3d8bf1dfb5833e9c131cf8c7b06a38c0"
server_key_version: "encryptionkey@0"
I20260928 21:54:19.848922 7051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:19.863224 7051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:54:19.865430 7051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:54:19.865553 7051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:54:19.865797 7051 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:54:19.866171 7051 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:54:19.866228 7051 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:19.866262 7051 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:54:19.866279 7051 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:19.872721 7051 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.66:34215
I20260928 21:54:19.872761 7178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.66:34215 every 8 connection(s)
I20260928 21:54:19.873025 7051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
Sep 28 21:54:19 dist-test-slave-k6lz krb5kdc[6812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632459, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.66@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:54:19.883220 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 7051
I20260928 21:54:19.883360 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/minicluster-data/ts-1/wal/instance
I20260928 21:54:19.883633 2777 external_mini_cluster.cc:1428] Setting key a65c2810cb521a9fe80519a89dfff5ea
I20260928 21:54:19.883747 6906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:19.873789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.66:38369 (local address 127.2.182.126:34355)
0928 21:54:19.873848 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:19.873850 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:19.874050 (+ 200us) server_negotiation.cc:407] Connection header received
0928 21:54:19.874558 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:19.874560 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:19.874571 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:19.874634 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:19.875678 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:19.876290 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:19.877413 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:19.877607 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:19.877678 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:19.880582 (+ 2904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:19.880584 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:19.880584 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:19.880593 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:19.881850 (+ 1257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:19.882234 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:19.882235 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:19.882236 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:19.882288 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:19.882515 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:19.882516 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:19.882516 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:19.882632 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:19.882699 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:19.883474 (+ 775us) server_negotiation.cc:299] Negotiation successful
0928 21:54:19.883651 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 21:54:19.883947 7181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:19.873724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:34355 (local address 127.2.182.66:38369)
0928 21:54:19.873920 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:19.873962 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:19.874449 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:19.874708 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:19.874711 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:19.874980 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:19.875543 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:19.875549 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:19.876399 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:19.876400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:19.877308 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:19.877311 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:19.877788 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:19.877789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:19.878026 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:19.878761 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:19.878771 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:19.880478 (+ 1707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:19.881931 (+ 1453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:19.881939 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:19.881942 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:19.882168 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:19.882349 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:19.882349 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:19.882350 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:19.882460 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:19.882799 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:19.882808 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:19.883153 (+ 345us) client_negotiation.cc:769] Sending connection context
0928 21:54:19.883314 (+ 161us) client_negotiation.cc:240] Negotiation successful
0928 21:54:19.883648 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":83,"threads_started":1}
I20260928 21:54:19.884397 7179 heartbeater.cc:347] Connected to a master server at 127.2.182.126:34355
I20260928 21:54:19.884493 7179 heartbeater.cc:464] Registering TS with master...
I20260928 21:54:19.884648 7179 heartbeater.cc:511] Master 127.2.182.126:34355 requested a full tablet report, sending...
I20260928 21:54:19.885035 6855 ts_manager.cc:194] Registered new tserver with Master: baba5f95f508446e927b82d746a50152 (127.2.182.66:34215)
I20260928 21:54:19.885448 6855 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.66@KRBTEST.COM'} at 127.2.182.66:38369
WARNING: no policy specified for kudu/127.2.182.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab.
Entry for principal kudu/127.2.182.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab.
I20260928 21:54:19.912190 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.67:0
--local_ip_for_outbound_sockets=127.2.182.67
--webserver_interface=127.2.182.67
--webserver_port=0
--tserver_master_addrs=127.2.182.126:34355
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab
--principal=kudu/127.2.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:41027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:54:19.995474 7184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:54:19.995695 7184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:54:19.995720 7184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:54:19.996922 7184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:54:19.996964 7184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:54:19.996991 7184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.67
I20260928 21:54:19.998539 7184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:41027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab
--principal=kudu/127.2.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.182.67
--webserver_port=0
--tserver_master_addrs=127.2.182.126:34355
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.67
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:54:19.998788 7184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:54:19.999001 7184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:54:20.004810 7184 server_base.cc:1034] running on GCE node
W20260928 21:54:20.005005 7192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:20.004772 7190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:20.004772 7189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:20.005515 7184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:20.005771 7184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:20.006974 7184 hybrid_clock.cc:648] HybridClock initialized: now 1790632460006894 us; error 93 us; skew 500 ppm
Sep 28 21:54:20 dist-test-slave-k6lz krb5kdc[6812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632460, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:54:20.010097 7184 init.cc:377] Logged in from keytab as kudu/127.2.182.67@KRBTEST.COM (short username kudu)
I20260928 21:54:20.010574 7184 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:20.010633 7184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:20.010739 7184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:54:20.011875 7184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/minicluster-data/ts-2/data/instance:
uuid: "e1231c1dc89e43d980ff5e47d2084549"
format_stamp: "Formatted at 2026-09-28 21:54:20 on dist-test-slave-k6lz"
server_key: "2225c772afc26525cd5ec143cb9f6f3c"
server_key_iv: "ca0942b773de4ee294905977882e2c9c"
server_key_version: "encryptionkey@0"
I20260928 21:54:20.012212 7184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/minicluster-data/ts-2/wal/instance:
uuid: "e1231c1dc89e43d980ff5e47d2084549"
format_stamp: "Formatted at 2026-09-28 21:54:20 on dist-test-slave-k6lz"
server_key: "2225c772afc26525cd5ec143cb9f6f3c"
server_key_iv: "ca0942b773de4ee294905977882e2c9c"
server_key_version: "encryptionkey@0"
I20260928 21:54:20.013767 7184 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:54:20.014770 7198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:20.015024 7184 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:54:20.015127 7184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/minicluster-data/ts-2/wal
uuid: "e1231c1dc89e43d980ff5e47d2084549"
format_stamp: "Formatted at 2026-09-28 21:54:20 on dist-test-slave-k6lz"
server_key: "2225c772afc26525cd5ec143cb9f6f3c"
server_key_iv: "ca0942b773de4ee294905977882e2c9c"
server_key_version: "encryptionkey@0"
I20260928 21:54:20.015226 7184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:20.040946 7184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:54:20.043944 7184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:54:20.044077 7184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:54:20.044277 7184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:54:20.044584 7184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:54:20.044623 7184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:20.044646 7184 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:54:20.044657 7184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:20.050630 7184 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.67:37253
I20260928 21:54:20.050680 7311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.67:37253 every 8 connection(s)
I20260928 21:54:20.050954 7184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
Sep 28 21:54:20 dist-test-slave-k6lz krb5kdc[6812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632460, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.67@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:54:20.057659 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 7184
I20260928 21:54:20.057833 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/minicluster-data/ts-2/wal/instance
I20260928 21:54:20.058048 2777 external_mini_cluster.cc:1428] Setting key 080fed5885e84f0fe774eb69e1b54516
I20260928 21:54:20.060689 6906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:20.051782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.67:59775 (local address 127.2.182.126:34355)
0928 21:54:20.051907 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:20.051909 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:20.051987 (+ 78us) server_negotiation.cc:407] Connection header received
0928 21:54:20.052476 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:20.052478 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:20.052491 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:20.052539 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:20.053525 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:20.054041 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:20.055068 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:20.055264 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:20.055324 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:20.057542 (+ 2218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:20.057544 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:20.057544 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:20.057551 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:20.059300 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:20.059737 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:20.059738 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:20.059739 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:20.059801 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:20.060021 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:20.060022 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:20.060022 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:20.060154 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:20.060210 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:20.060494 (+ 284us) server_negotiation.cc:299] Negotiation successful
0928 21:54:20.060596 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260928 21:54:20.060843 7314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:20.051707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:34355 (local address 127.2.182.67:59775)
0928 21:54:20.051866 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:20.051909 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:20.052372 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:20.052610 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:20.052614 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:20.052909 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:20.053426 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:20.053431 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:20.054130 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:20.054131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:20.054976 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:20.054978 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:20.055361 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:20.055362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:20.055482 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:20.055988 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:20.055997 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:20.057436 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:20.059413 (+ 1977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:20.059422 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:20.059425 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:20.059661 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:20.059859 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:20.059859 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:20.059860 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:20.059965 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:20.060216 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:20.060222 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:20.060364 (+ 142us) client_negotiation.cc:769] Sending connection context
0928 21:54:20.060455 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 21:54:20.060619 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":72,"threads_started":1}
I20260928 21:54:20.061323 7312 heartbeater.cc:347] Connected to a master server at 127.2.182.126:34355
I20260928 21:54:20.061417 7312 heartbeater.cc:464] Registering TS with master...
I20260928 21:54:20.061566 7312 heartbeater.cc:511] Master 127.2.182.126:34355 requested a full tablet report, sending...
I20260928 21:54:20.061965 6855 ts_manager.cc:194] Registered new tserver with Master: e1231c1dc89e43d980ff5e47d2084549 (127.2.182.67:37253)
I20260928 21:54:20.062378 6855 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.67@KRBTEST.COM'} at 127.2.182.67:59775
I20260928 21:54:20.070238 2777 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:54:20.078146 6906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:20.071591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59406 (local address 127.2.182.126:34355)
0928 21:54:20.071663 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:20.071665 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:20.071747 (+ 82us) server_negotiation.cc:407] Connection header received
0928 21:54:20.071790 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:20.071791 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:20.071804 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:20.071849 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:20.072401 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:20.072964 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:20.073729 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:20.073918 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:20.074023 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:20.075547 (+ 1524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:20.075548 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:20.075549 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:20.075556 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:20.076961 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:20.077291 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:20.077292 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:20.077293 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:20.077348 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:20.077516 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:20.077516 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:20.077517 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:20.077722 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:20.077824 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:20.077961 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 21:54:20.078047 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 21:54:20.080081 6855 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59406:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:54:20.081919 6855 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:54:20.088205 7328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:20.084626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50146 (local address 127.2.182.65:37043)
0928 21:54:20.084878 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:20.084883 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:20.084894 (+ 11us) server_negotiation.cc:407] Connection header received
0928 21:54:20.084957 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:20.084959 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:20.084974 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:20.085043 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:54:20.085191 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:20.086031 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:20.087128 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:20.087823 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:20.087888 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:20.087957 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:20.088079 (+ 122us) server_negotiation.cc:299] Negotiation successful
0928 21:54:20.088117 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":68,"threads_started":1}
I20260928 21:54:20.088200 7324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:20.084531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.65:37043 (local address 127.0.0.1:50146)
0928 21:54:20.084764 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:20.084806 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:20.084876 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:20.085053 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:20.085056 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:20.085058 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:54:20.085135 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:20.085137 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:20.086115 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:20.086116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:20.087040 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:20.087043 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:20.087892 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:20.087894 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:20.087982 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:20.087996 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 21:54:20.088060 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 21:54:20.088117 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"mutex_wait_us":61,"thread_start_us":59,"threads_started":1}
I20260928 21:54:20.088697 7323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:20.084349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.66:34215 (local address 127.0.0.1:52258)
0928 21:54:20.084762 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:20.084806 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:20.084875 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:20.085091 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:20.085092 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:20.085094 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:54:20.085148 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:20.085150 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:20.086148 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:20.086149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:20.087437 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:20.087440 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:20.088480 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:20.088481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:20.088547 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:20.088551 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:54:20.088577 (+ 26us) client_negotiation.cc:240] Negotiation successful
0928 21:54:20.088613 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"mutex_wait_us":26,"thread_start_us":71,"threads_started":1}
I20260928 21:54:20.088886 6980 tablet_service.cc:1467] Processing CreateTablet for tablet 89e7097caef04dc98ebfc25a3813f3c7 (DEFAULT_TABLE table=test-table [id=a8051e61a31342db81568b6e31f8ab0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:54:20.089109 7325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:20.084440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52258 (local address 127.2.182.66:34215)
0928 21:54:20.084805 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:20.084810 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:20.084962 (+ 152us) server_negotiation.cc:407] Connection header received
0928 21:54:20.085006 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:20.085009 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:20.085023 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:20.085083 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:54:20.085200 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:20.086056 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:20.087531 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:20.088400 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:20.088876 (+ 476us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:20.088948 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:20.088976 (+ 28us) server_negotiation.cc:299] Negotiation successful
0928 21:54:20.089017 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":82,"threads_started":1}
I20260928 21:54:20.089253 6980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89e7097caef04dc98ebfc25a3813f3c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:54:20.089728 7113 tablet_service.cc:1467] Processing CreateTablet for tablet 89e7097caef04dc98ebfc25a3813f3c7 (DEFAULT_TABLE table=test-table [id=a8051e61a31342db81568b6e31f8ab0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:54:20.090090 7113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89e7097caef04dc98ebfc25a3813f3c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:54:20.092907 7326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:20.084349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.67:37253 (local address 127.0.0.1:48004)
0928 21:54:20.084762 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:20.084806 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:20.084875 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:20.086420 (+ 1545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:20.086420 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:20.086423 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:54:20.086460 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:20.086462 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:20.087691 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:20.087691 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:20.088908 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:20.088911 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:20.092550 (+ 3639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:20.092551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:20.092629 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:20.092635 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 21:54:20.092694 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 21:54:20.092744 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":50,"threads_started":1}
I20260928 21:54:20.092949 7327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:20.084602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48004 (local address 127.2.182.67:37253)
0928 21:54:20.086194 (+ 1592us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:20.086199 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:20.086208 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:54:20.086244 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:20.086246 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:20.086262 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:20.086332 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:54:20.086531 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:20.087599 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:20.088976 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:20.092401 (+ 3425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:20.092506 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:20.092565 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:20.092713 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 21:54:20.092760 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1549,"thread_start_us":79,"threads_started":1}
I20260928 21:54:20.093562 7329 tablet_bootstrap.cc:492] T 89e7097caef04dc98ebfc25a3813f3c7 P 4dfc8804c61742f7b54798c0b0533d98: Bootstrap starting.
I20260928 21:54:20.094259 7246 tablet_service.cc:1467] Processing CreateTablet for tablet 89e7097caef04dc98ebfc25a3813f3c7 (DEFAULT_TABLE table=test-table [id=a8051e61a31342db81568b6e31f8ab0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:54:20.094543 7329 tablet_bootstrap.cc:654] T 89e7097caef04dc98ebfc25a3813f3c7 P 4dfc8804c61742f7b54798c0b0533d98: Neither blocks nor log segments found. Creating new log.
I20260928 21:54:20.094429 7330 tablet_bootstrap.cc:492] T 89e7097caef04dc98ebfc25a3813f3c7 P baba5f95f508446e927b82d746a50152: Bootstrap starting.
I20260928 21:54:20.094686 7246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89e7097caef04dc98ebfc25a3813f3c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:54:20.095074 7329 log.cc:824] T 89e7097caef04dc98ebfc25a3813f3c7 P 4dfc8804c61742f7b54798c0b0533d98: Log is configured to *not* fsync() on all Append() calls
I20260928 21:54:20.095314 7330 tablet_bootstrap.cc:654] T 89e7097caef04dc98ebfc25a3813f3c7 P baba5f95f508446e927b82d746a50152: Neither blocks nor log segments found. Creating new log.
I20260928 21:54:20.095927 7329 tablet_bootstrap.cc:492] T 89e7097caef04dc98ebfc25a3813f3c7 P 4dfc8804c61742f7b54798c0b0533d98: No bootstrap required, opened a new log
I20260928 21:54:20.095992 7329 ts_tablet_manager.cc:1397] T 89e7097caef04dc98ebfc25a3813f3c7 P 4dfc8804c61742f7b54798c0b0533d98: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 21:54:20.096067 7330 log.cc:824] T 89e7097caef04dc98ebfc25a3813f3c7 P baba5f95f508446e927b82d746a50152: Log is configured to *not* fsync() on all Append() calls
I20260928 21:54:20.096907 7330 tablet_bootstrap.cc:492] T 89e7097caef04dc98ebfc25a3813f3c7 P baba5f95f508446e927b82d746a50152: No bootstrap required, opened a new log
I20260928 21:54:20.096980 7330 ts_tablet_manager.cc:1397] T 89e7097caef04dc98ebfc25a3813f3c7 P baba5f95f508446e927b82d746a50152: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260928 21:54:20.097675 7329 raft_consensus.cc:348] T 89e7097caef04dc98ebfc25a3813f3c7 P 4dfc8804c61742f7b54798c0b0533d98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baba5f95f508446e927b82d746a50152" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 34215 } } peers { permanent_uuid: "e1231c1dc89e43d980ff5e47d2084549" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 37253 } } peers { permanent_uuid: "4dfc8804c61742f7b54798c0b0533d98" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 37043 } }
I20260928 21:54:20.097826 7329 raft_consensus.cc:374] T 89e7097caef04dc98ebfc25a3813f3c7 P 4dfc8804c61742f7b54798c0b0533d98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:54:20.097859 7329 raft_consensus.cc:729] T 89e7097caef04dc98ebfc25a3813f3c7 P 4dfc8804c61742f7b54798c0b0533d98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dfc8804c61742f7b54798c0b0533d98, State: Initialized, Role: FOLLOWER
I20260928 21:54:20.097982 7329 consensus_queue.cc:260] T 89e7097caef04dc98ebfc25a3813f3c7 P 4dfc8804c61742f7b54798c0b0533d98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baba5f95f508446e927b82d746a50152" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 34215 } } peers { permanent_uuid: "e1231c1dc89e43d980ff5e47d2084549" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 37253 } } peers { permanent_uuid: "4dfc8804c61742f7b54798c0b0533d98" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 37043 } }
I20260928 21:54:20.098268 7329 ts_tablet_manager.cc:1428] T 89e7097caef04dc98ebfc25a3813f3c7 P 4dfc8804c61742f7b54798c0b0533d98: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 21:54:20.098387 7046 heartbeater.cc:503] Master 127.2.182.126:34355 was elected leader, sending a full tablet report...
I20260928 21:54:20.098253 7333 tablet_bootstrap.cc:492] T 89e7097caef04dc98ebfc25a3813f3c7 P e1231c1dc89e43d980ff5e47d2084549: Bootstrap starting.
I20260928 21:54:20.098925 7330 raft_consensus.cc:348] T 89e7097caef04dc98ebfc25a3813f3c7 P baba5f95f508446e927b82d746a50152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baba5f95f508446e927b82d746a50152" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 34215 } } peers { permanent_uuid: "e1231c1dc89e43d980ff5e47d2084549" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 37253 } } peers { permanent_uuid: "4dfc8804c61742f7b54798c0b0533d98" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 37043 } }
I20260928 21:54:20.099078 7330 raft_consensus.cc:374] T 89e7097caef04dc98ebfc25a3813f3c7 P baba5f95f508446e927b82d746a50152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:54:20.099112 7330 raft_consensus.cc:729] T 89e7097caef04dc98ebfc25a3813f3c7 P baba5f95f508446e927b82d746a50152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baba5f95f508446e927b82d746a50152, State: Initialized, Role: FOLLOWER
I20260928 21:54:20.099246 7330 consensus_queue.cc:260] T 89e7097caef04dc98ebfc25a3813f3c7 P baba5f95f508446e927b82d746a50152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baba5f95f508446e927b82d746a50152" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 34215 } } peers { permanent_uuid: "e1231c1dc89e43d980ff5e47d2084549" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 37253 } } peers { permanent_uuid: "4dfc8804c61742f7b54798c0b0533d98" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 37043 } }
I20260928 21:54:20.099548 7330 ts_tablet_manager.cc:1428] T 89e7097caef04dc98ebfc25a3813f3c7 P baba5f95f508446e927b82d746a50152: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260928 21:54:20.099588 7179 heartbeater.cc:503] Master 127.2.182.126:34355 was elected leader, sending a full tablet report...
I20260928 21:54:20.099756 7333 tablet_bootstrap.cc:654] T 89e7097caef04dc98ebfc25a3813f3c7 P e1231c1dc89e43d980ff5e47d2084549: Neither blocks nor log segments found. Creating new log.
I20260928 21:54:20.100272 7333 log.cc:824] T 89e7097caef04dc98ebfc25a3813f3c7 P e1231c1dc89e43d980ff5e47d2084549: Log is configured to *not* fsync() on all Append() calls
I20260928 21:54:20.100939 7333 tablet_bootstrap.cc:492] T 89e7097caef04dc98ebfc25a3813f3c7 P e1231c1dc89e43d980ff5e47d2084549: No bootstrap required, opened a new log
I20260928 21:54:20.101001 7333 ts_tablet_manager.cc:1397] T 89e7097caef04dc98ebfc25a3813f3c7 P e1231c1dc89e43d980ff5e47d2084549: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 21:54:20.102394 7333 raft_consensus.cc:348] T 89e7097caef04dc98ebfc25a3813f3c7 P e1231c1dc89e43d980ff5e47d2084549 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baba5f95f508446e927b82d746a50152" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 34215 } } peers { permanent_uuid: "e1231c1dc89e43d980ff5e47d2084549" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 37253 } } peers { permanent_uuid: "4dfc8804c61742f7b54798c0b0533d98" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 37043 } }
I20260928 21:54:20.102530 7333 raft_consensus.cc:374] T 89e7097caef04dc98ebfc25a3813f3c7 P e1231c1dc89e43d980ff5e47d2084549 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:54:20.102560 7333 raft_consensus.cc:729] T 89e7097caef04dc98ebfc25a3813f3c7 P e1231c1dc89e43d980ff5e47d2084549 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1231c1dc89e43d980ff5e47d2084549, State: Initialized, Role: FOLLOWER
I20260928 21:54:20.102674 7333 consensus_queue.cc:260] T 89e7097caef04dc98ebfc25a3813f3c7 P e1231c1dc89e43d980ff5e47d2084549 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baba5f95f508446e927b82d746a50152" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 34215 } } peers { permanent_uuid: "e1231c1dc89e43d980ff5e47d2084549" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 37253 } } peers { permanent_uuid: "4dfc8804c61742f7b54798c0b0533d98" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 37043 } }
I20260928 21:54:20.102949 7333 ts_tablet_manager.cc:1428] T 89e7097caef04dc98ebfc25a3813f3c7 P e1231c1dc89e43d980ff5e47d2084549: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 21:54:20.103034 7312 heartbeater.cc:503] Master 127.2.182.126:34355 was elected leader, sending a full tablet report...
W20260928 21:54:20.123517 7180 tablet.cc:2369] T 89e7097caef04dc98ebfc25a3813f3c7 P baba5f95f508446e927b82d746a50152: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 21:54:20.214458 7047 tablet.cc:2369] T 89e7097caef04dc98ebfc25a3813f3c7 P 4dfc8804c61742f7b54798c0b0533d98: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 21:54:20.301532 7313 tablet.cc:2369] T 89e7097caef04dc98ebfc25a3813f3c7 P e1231c1dc89e43d980ff5e47d2084549: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:54:20.378067 7334 raft_consensus.cc:482] T 89e7097caef04dc98ebfc25a3813f3c7 P 4dfc8804c61742f7b54798c0b0533d98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:54:20.378192 7334 raft_consensus.cc:504] T 89e7097caef04dc98ebfc25a3813f3c7 P 4dfc8804c61742f7b54798c0b0533d98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baba5f95f508446e927b82d746a50152" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 34215 } } peers { permanent_uuid: "e1231c1dc89e43d980ff5e47d2084549" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 37253 } } peers { permanent_uuid: "4dfc8804c61742f7b54798c0b0533d98" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 37043 } }
I20260928 21:54:20.378556 7334 leader_election.cc:288] T 89e7097caef04dc98ebfc25a3813f3c7 P 4dfc8804c61742f7b54798c0b0533d98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers baba5f95f508446e927b82d746a50152 (127.2.182.66:34215), e1231c1dc89e43d980ff5e47d2084549 (127.2.182.67:37253)
I20260928 21:54:20.381474 7327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:20.378989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.65:39649 (local address 127.2.182.67:37253)
0928 21:54:20.379029 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:20.379031 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:20.379038 (+ 7us) server_negotiation.cc:407] Connection header received
0928 21:54:20.379064 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:20.379064 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:20.379073 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:20.379126 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:54:20.379317 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:20.379783 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:20.380764 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:20.381137 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:20.381203 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:20.381212 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:20.381370 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 21:54:20.381403 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 21:54:20.381474 7325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:20.378874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.65:33441 (local address 127.2.182.66:34215)
0928 21:54:20.378919 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:20.378922 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:20.378935 (+ 13us) server_negotiation.cc:407] Connection header received
0928 21:54:20.379003 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:20.379004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:20.379013 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:20.379058 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:54:20.379256 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:20.379781 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:20.380782 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:20.381148 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:20.381222 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:20.381231 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:20.381367 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 21:54:20.381402 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 21:54:20.381525 7338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:20.378730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.66:34215 (local address 127.2.182.65:33441)
0928 21:54:20.378874 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:20.378886 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:20.378945 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:20.379073 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:20.379073 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:20.379075 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:54:20.379147 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:20.379149 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:20.379865 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:20.379866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:20.380692 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:20.380695 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:20.381239 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:20.381239 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:20.381301 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:20.381305 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:54:20.381350 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 21:54:20.381394 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":87,"threads_started":1}
I20260928 21:54:20.381524 7339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:20.378805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.67:37253 (local address 127.2.182.65:39649)
0928 21:54:20.378916 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:20.378927 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:20.378982 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:20.379145 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:20.379145 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:20.379147 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:54:20.379205 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:20.379206 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:20.379860 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:20.379861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:20.380673 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:20.380676 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:20.381219 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:20.381220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:20.381302 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:20.381305 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:54:20.381354 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 21:54:20.381397 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":67,"threads_started":1}
I20260928 21:54:20.382066 7131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89e7097caef04dc98ebfc25a3813f3c7" candidate_uuid: "4dfc8804c61742f7b54798c0b0533d98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baba5f95f508446e927b82d746a50152" is_pre_election: true
I20260928 21:54:20.382259 7131 raft_consensus.cc:2454] T 89e7097caef04dc98ebfc25a3813f3c7 P baba5f95f508446e927b82d746a50152 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dfc8804c61742f7b54798c0b0533d98 in term 0.
I20260928 21:54:20.382345 7266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89e7097caef04dc98ebfc25a3813f3c7" candidate_uuid: "4dfc8804c61742f7b54798c0b0533d98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1231c1dc89e43d980ff5e47d2084549" is_pre_election: true
I20260928 21:54:20.382541 7266 raft_consensus.cc:2454] T 89e7097caef04dc98ebfc25a3813f3c7 P e1231c1dc89e43d980ff5e47d2084549 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dfc8804c61742f7b54798c0b0533d98 in term 0.
I20260928 21:54:20.382695 6935 leader_election.cc:302] T 89e7097caef04dc98ebfc25a3813f3c7 P 4dfc8804c61742f7b54798c0b0533d98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4dfc8804c61742f7b54798c0b0533d98, baba5f95f508446e927b82d746a50152; no voters:
I20260928 21:54:20.382858 7334 raft_consensus.cc:2781] T 89e7097caef04dc98ebfc25a3813f3c7 P 4dfc8804c61742f7b54798c0b0533d98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:54:20.382925 7334 raft_consensus.cc:482] T 89e7097caef04dc98ebfc25a3813f3c7 P 4dfc8804c61742f7b54798c0b0533d98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:54:20.382948 7334 raft_consensus.cc:3037] T 89e7097caef04dc98ebfc25a3813f3c7 P 4dfc8804c61742f7b54798c0b0533d98 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:54:20.383666 7334 raft_consensus.cc:504] T 89e7097caef04dc98ebfc25a3813f3c7 P 4dfc8804c61742f7b54798c0b0533d98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baba5f95f508446e927b82d746a50152" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 34215 } } peers { permanent_uuid: "e1231c1dc89e43d980ff5e47d2084549" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 37253 } } peers { permanent_uuid: "4dfc8804c61742f7b54798c0b0533d98" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 37043 } }
I20260928 21:54:20.383812 7334 leader_election.cc:288] T 89e7097caef04dc98ebfc25a3813f3c7 P 4dfc8804c61742f7b54798c0b0533d98 [CANDIDATE]: Term 1 election: Requested vote from peers baba5f95f508446e927b82d746a50152 (127.2.182.66:34215), e1231c1dc89e43d980ff5e47d2084549 (127.2.182.67:37253)
I20260928 21:54:20.383980 7131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89e7097caef04dc98ebfc25a3813f3c7" candidate_uuid: "4dfc8804c61742f7b54798c0b0533d98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baba5f95f508446e927b82d746a50152"
I20260928 21:54:20.384042 7266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89e7097caef04dc98ebfc25a3813f3c7" candidate_uuid: "4dfc8804c61742f7b54798c0b0533d98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1231c1dc89e43d980ff5e47d2084549"
I20260928 21:54:20.384063 7131 raft_consensus.cc:3037] T 89e7097caef04dc98ebfc25a3813f3c7 P baba5f95f508446e927b82d746a50152 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:54:20.384130 7266 raft_consensus.cc:3037] T 89e7097caef04dc98ebfc25a3813f3c7 P e1231c1dc89e43d980ff5e47d2084549 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:54:20.384779 7131 raft_consensus.cc:2454] T 89e7097caef04dc98ebfc25a3813f3c7 P baba5f95f508446e927b82d746a50152 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4dfc8804c61742f7b54798c0b0533d98 in term 1.
I20260928 21:54:20.384922 7266 raft_consensus.cc:2454] T 89e7097caef04dc98ebfc25a3813f3c7 P e1231c1dc89e43d980ff5e47d2084549 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4dfc8804c61742f7b54798c0b0533d98 in term 1.
I20260928 21:54:20.384951 6935 leader_election.cc:302] T 89e7097caef04dc98ebfc25a3813f3c7 P 4dfc8804c61742f7b54798c0b0533d98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4dfc8804c61742f7b54798c0b0533d98, baba5f95f508446e927b82d746a50152; no voters:
I20260928 21:54:20.385119 7334 raft_consensus.cc:2781] T 89e7097caef04dc98ebfc25a3813f3c7 P 4dfc8804c61742f7b54798c0b0533d98 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:54:20.385272 7334 raft_consensus.cc:686] T 89e7097caef04dc98ebfc25a3813f3c7 P 4dfc8804c61742f7b54798c0b0533d98 [term 1 LEADER]: Becoming Leader. State: Replica: 4dfc8804c61742f7b54798c0b0533d98, State: Running, Role: LEADER
I20260928 21:54:20.385377 7334 consensus_queue.cc:237] T 89e7097caef04dc98ebfc25a3813f3c7 P 4dfc8804c61742f7b54798c0b0533d98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baba5f95f508446e927b82d746a50152" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 34215 } } peers { permanent_uuid: "e1231c1dc89e43d980ff5e47d2084549" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 37253 } } peers { permanent_uuid: "4dfc8804c61742f7b54798c0b0533d98" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 37043 } }
I20260928 21:54:20.386132 6855 catalog_manager.cc:5495] T 89e7097caef04dc98ebfc25a3813f3c7 P 4dfc8804c61742f7b54798c0b0533d98 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4dfc8804c61742f7b54798c0b0533d98 (127.2.182.65). New cstate: current_term: 1 leader_uuid: "4dfc8804c61742f7b54798c0b0533d98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baba5f95f508446e927b82d746a50152" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 34215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1231c1dc89e43d980ff5e47d2084549" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 37253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4dfc8804c61742f7b54798c0b0533d98" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 37043 } health_report { overall_health: HEALTHY } } }
I20260928 21:54:20.446698 7328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:20.444332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50148 (local address 127.2.182.65:37043)
0928 21:54:20.444396 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:20.444397 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:20.444405 (+ 8us) server_negotiation.cc:407] Connection header received
0928 21:54:20.444439 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:20.444440 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:20.444457 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:20.444517 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:54:20.444633 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:20.445119 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:20.445924 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:20.446109 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:20.446162 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:20.446382 (+ 220us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:54:20.446491 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:54:20.446565 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:20.446615 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 21:54:20.446635 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 21:54:20.448642 7266 raft_consensus.cc:1261] T 89e7097caef04dc98ebfc25a3813f3c7 P e1231c1dc89e43d980ff5e47d2084549 [term 1 FOLLOWER]: Refusing update from remote peer 4dfc8804c61742f7b54798c0b0533d98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:54:20.448642 7131 raft_consensus.cc:1261] T 89e7097caef04dc98ebfc25a3813f3c7 P baba5f95f508446e927b82d746a50152 [term 1 FOLLOWER]: Refusing update from remote peer 4dfc8804c61742f7b54798c0b0533d98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:54:20.449095 7340 consensus_queue.cc:1035] T 89e7097caef04dc98ebfc25a3813f3c7 P 4dfc8804c61742f7b54798c0b0533d98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "baba5f95f508446e927b82d746a50152" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 34215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:54:20.449236 7334 consensus_queue.cc:1035] T 89e7097caef04dc98ebfc25a3813f3c7 P 4dfc8804c61742f7b54798c0b0533d98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1231c1dc89e43d980ff5e47d2084549" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 37253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:54:20.452014 7345 mvcc.cc:205] Tried to move back new op lower bound from 7334430557994774528 to 7334430557739040768. Current Snapshot: MvccSnapshot[applied={T|T < 7334430557994774528}]
I20260928 21:54:20.452749 7344 mvcc.cc:205] Tried to move back new op lower bound from 7334430557994774528 to 7334430557739040768. Current Snapshot: MvccSnapshot[applied={T|T < 7334430557994774528}]
I20260928 21:54:20.455181 6856 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59406:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:54:20.455287 6856 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59406:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:54:20.456055 6856 catalog_manager.cc:5782] T 00000000000000000000000000000000 P df90d7d076d24d03beddac0168792541: Sending DeleteTablet for 3 replicas of tablet 89e7097caef04dc98ebfc25a3813f3c7
I20260928 21:54:20.456380 7113 tablet_service.cc:1514] Processing DeleteTablet for tablet 89e7097caef04dc98ebfc25a3813f3c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:54:20 UTC) from {username='kudu', principal='kudu/127.2.182.126@KRBTEST.COM'} at 127.0.0.1:52258
I20260928 21:54:20.456403 6980 tablet_service.cc:1514] Processing DeleteTablet for tablet 89e7097caef04dc98ebfc25a3813f3c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:54:20 UTC) from {username='kudu', principal='kudu/127.2.182.126@KRBTEST.COM'} at 127.0.0.1:50146
I20260928 21:54:20.456691 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 6918
I20260928 21:54:20.456704 7246 tablet_service.cc:1514] Processing DeleteTablet for tablet 89e7097caef04dc98ebfc25a3813f3c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:54:20 UTC) from {username='kudu', principal='kudu/127.2.182.126@KRBTEST.COM'} at 127.0.0.1:48004
I20260928 21:54:20.457893 7354 tablet_replica.cc:330] T 89e7097caef04dc98ebfc25a3813f3c7 P e1231c1dc89e43d980ff5e47d2084549: stopping tablet replica
I20260928 21:54:20.458015 7354 raft_consensus.cc:2229] T 89e7097caef04dc98ebfc25a3813f3c7 P e1231c1dc89e43d980ff5e47d2084549 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:54:20.458132 7354 raft_consensus.cc:2258] T 89e7097caef04dc98ebfc25a3813f3c7 P e1231c1dc89e43d980ff5e47d2084549 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:54:20.458300 7352 tablet_replica.cc:330] T 89e7097caef04dc98ebfc25a3813f3c7 P baba5f95f508446e927b82d746a50152: stopping tablet replica
I20260928 21:54:20.458416 7352 raft_consensus.cc:2229] T 89e7097caef04dc98ebfc25a3813f3c7 P baba5f95f508446e927b82d746a50152 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:54:20.458527 7352 raft_consensus.cc:2258] T 89e7097caef04dc98ebfc25a3813f3c7 P baba5f95f508446e927b82d746a50152 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:54:20.458678 7354 ts_tablet_manager.cc:1905] T 89e7097caef04dc98ebfc25a3813f3c7 P e1231c1dc89e43d980ff5e47d2084549: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:54:20.458961 7352 ts_tablet_manager.cc:1905] T 89e7097caef04dc98ebfc25a3813f3c7 P baba5f95f508446e927b82d746a50152: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:54:20.460673 7354 ts_tablet_manager.cc:1918] T 89e7097caef04dc98ebfc25a3813f3c7 P e1231c1dc89e43d980ff5e47d2084549: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:54:20.460757 7354 log.cc:1196] T 89e7097caef04dc98ebfc25a3813f3c7 P e1231c1dc89e43d980ff5e47d2084549: Deleting WAL directory at /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/minicluster-data/ts-2/wal/wals/89e7097caef04dc98ebfc25a3813f3c7
I20260928 21:54:20.460798 7352 ts_tablet_manager.cc:1918] T 89e7097caef04dc98ebfc25a3813f3c7 P baba5f95f508446e927b82d746a50152: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:54:20.460879 7352 log.cc:1196] T 89e7097caef04dc98ebfc25a3813f3c7 P baba5f95f508446e927b82d746a50152: Deleting WAL directory at /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632440055392-2777-0/minicluster-data/ts-1/wal/wals/89e7097caef04dc98ebfc25a3813f3c7
I20260928 21:54:20.461011 7354 ts_tablet_manager.cc:1939] T 89e7097caef04dc98ebfc25a3813f3c7 P e1231c1dc89e43d980ff5e47d2084549: Deleting consensus metadata
I20260928 21:54:20.461118 7352 ts_tablet_manager.cc:1939] T 89e7097caef04dc98ebfc25a3813f3c7 P baba5f95f508446e927b82d746a50152: Deleting consensus metadata
I20260928 21:54:20.461498 6840 catalog_manager.cc:4841] TS e1231c1dc89e43d980ff5e47d2084549 (127.2.182.67:37253): tablet 89e7097caef04dc98ebfc25a3813f3c7 (table test-table [id=a8051e61a31342db81568b6e31f8ab0e]) successfully deleted
I20260928 21:54:20.461526 6842 catalog_manager.cc:4841] TS baba5f95f508446e927b82d746a50152 (127.2.182.66:34215): tablet 89e7097caef04dc98ebfc25a3813f3c7 (table test-table [id=a8051e61a31342db81568b6e31f8ab0e]) successfully deleted
W20260928 21:54:20.462656 6842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.182.65:37043 (error 108)
I20260928 21:54:20.462924 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 7051
I20260928 21:54:20.463876 7326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:20.463719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.65:37043 (local address 127.0.0.1:50158)
0928 21:54:20.463789 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:20.463838 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.65:37043: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260928 21:54:20.463999 6842 catalog_manager.cc:4584] TS 4dfc8804c61742f7b54798c0b0533d98 (127.2.182.65:37043): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 89e7097caef04dc98ebfc25a3813f3c7: Network error: Client connection negotiation failed: client connection to 127.2.182.65:37043: connect: Connection refused (error 111)
I20260928 21:54:20.467618 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 7184
I20260928 21:54:20.471424 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 6825
2026-09-28T21:54:20Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2205 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-28T21:54:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:54:20Z Disabled control of system clock
I20260928 21:54:20.497639 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632440055392-2777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632440055392-2777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632440055392-2777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:43003
--webserver_interface=127.2.182.126
--webserver_port=0
--builtin_ntp_servers=127.2.182.84:34099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.126:43003
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 21:54:20.580713 7359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:54:20.580973 7359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:54:20.581007 7359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:54:20.582324 7359 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:54:20.582381 7359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:54:20.582396 7359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:54:20.582413 7359 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:54:20.582429 7359 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:54:20.584335 7359 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:34099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632440055392-2777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632440055392-2777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.126:43003
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:43003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632440055392-2777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:54:20.584669 7359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:54:20.584985 7359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:54:20.590287 7367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:20.590413 7365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:20.590652 7364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:20.590806 7359 server_base.cc:1034] running on GCE node
I20260928 21:54:20.591228 7359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:20.591611 7359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:20.592861 7359 hybrid_clock.cc:648] HybridClock initialized: now 1790632460592838 us; error 42 us; skew 500 ppm
I20260928 21:54:20.594244 7359 webserver.cc:460] Webserver started at http://127.2.182.126:40147/ using document root <none> and password file <none>
I20260928 21:54:20.594467 7359 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:20.594516 7359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:20.594643 7359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:54:20.595805 7359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632440055392-2777-0/minicluster-data/master-0/data/instance:
uuid: "78a6aac4a0a2423e95328deb5807f787"
format_stamp: "Formatted at 2026-09-28 21:54:20 on dist-test-slave-k6lz"
server_key: "1f0be31c7e1202de2cf4d5216f2f9965"
server_key_iv: "4bd11d4ad10fcf71b2cc2a382953ed2f"
server_key_version: "encryptionkey@0"
I20260928 21:54:20.596150 7359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632440055392-2777-0/minicluster-data/master-0/wal/instance:
uuid: "78a6aac4a0a2423e95328deb5807f787"
format_stamp: "Formatted at 2026-09-28 21:54:20 on dist-test-slave-k6lz"
server_key: "1f0be31c7e1202de2cf4d5216f2f9965"
server_key_iv: "4bd11d4ad10fcf71b2cc2a382953ed2f"
server_key_version: "encryptionkey@0"
I20260928 21:54:20.597492 7359 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:54:20.598315 7373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:20.598529 7359 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:54:20.598627 7359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632440055392-2777-0/minicluster-data/master-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632440055392-2777-0/minicluster-data/master-0/wal
uuid: "78a6aac4a0a2423e95328deb5807f787"
format_stamp: "Formatted at 2026-09-28 21:54:20 on dist-test-slave-k6lz"
server_key: "1f0be31c7e1202de2cf4d5216f2f9965"
server_key_iv: "4bd11d4ad10fcf71b2cc2a382953ed2f"
server_key_version: "encryptionkey@0"
I20260928 21:54:20.598690 7359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632440055392-2777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632440055392-2777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632440055392-2777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:20.602264 7359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:54:20.602548 7359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:54:20.602669 7359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:54:20.606808 7359 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.126:43003
I20260928 21:54:20.606873 7425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.126:43003 every 8 connection(s)
I20260928 21:54:20.607192 7359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
I20260928 21:54:20.607854 7426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:54:20.610631 7426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78a6aac4a0a2423e95328deb5807f787: Bootstrap starting.
I20260928 21:54:20.611184 7426 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 78a6aac4a0a2423e95328deb5807f787: Neither blocks nor log segments found. Creating new log.
I20260928 21:54:20.611620 7426 log.cc:824] T 00000000000000000000000000000000 P 78a6aac4a0a2423e95328deb5807f787: Log is configured to *not* fsync() on all Append() calls
I20260928 21:54:20.612030 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 7359
I20260928 21:54:20.612131 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632440055392-2777-0/minicluster-data/master-0/wal/instance
I20260928 21:54:20.612299 2777 external_mini_cluster.cc:1428] Setting key 3521c936543828f406deff0b4505b34f
I20260928 21:54:20.612584 7426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78a6aac4a0a2423e95328deb5807f787: No bootstrap required, opened a new log
I20260928 21:54:20.615275 7426 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78a6aac4a0a2423e95328deb5807f787 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78a6aac4a0a2423e95328deb5807f787" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43003 } }
I20260928 21:54:20.615641 7426 raft_consensus.cc:374] T 00000000000000000000000000000000 P 78a6aac4a0a2423e95328deb5807f787 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:54:20.615708 7426 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78a6aac4a0a2423e95328deb5807f787 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78a6aac4a0a2423e95328deb5807f787, State: Initialized, Role: FOLLOWER
I20260928 21:54:20.616379 7426 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78a6aac4a0a2423e95328deb5807f787 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78a6aac4a0a2423e95328deb5807f787" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43003 } }
I20260928 21:54:20.616525 7426 raft_consensus.cc:388] T 00000000000000000000000000000000 P 78a6aac4a0a2423e95328deb5807f787 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:54:20.616564 7426 raft_consensus.cc:482] T 00000000000000000000000000000000 P 78a6aac4a0a2423e95328deb5807f787 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:54:20.616604 7426 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78a6aac4a0a2423e95328deb5807f787 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:54:20.617316 7430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:20.612861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33982 (local address 127.2.182.126:43003)
0928 21:54:20.613345 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:20.613353 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:20.613383 (+ 30us) server_negotiation.cc:407] Connection header received
0928 21:54:20.615194 (+ 1811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:20.615205 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:20.615270 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:20.615510 (+ 240us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:20.616592 (+ 1082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:20.616597 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:20.616597 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:54:20.616602 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:20.616655 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:54:20.616673 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:20.616755 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:20.616831 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 21:54:20.616909 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":87,"threads_started":1}
I20260928 21:54:20.617381 7426 raft_consensus.cc:504] T 00000000000000000000000000000000 P 78a6aac4a0a2423e95328deb5807f787 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78a6aac4a0a2423e95328deb5807f787" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43003 } }
I20260928 21:54:20.617497 7426 leader_election.cc:302] T 00000000000000000000000000000000 P 78a6aac4a0a2423e95328deb5807f787 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 78a6aac4a0a2423e95328deb5807f787; no voters:
I20260928 21:54:20.617668 7426 leader_election.cc:288] T 00000000000000000000000000000000 P 78a6aac4a0a2423e95328deb5807f787 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:54:20.617712 7431 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 78a6aac4a0a2423e95328deb5807f787 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:54:20.617847 7431 raft_consensus.cc:686] T 00000000000000000000000000000000 P 78a6aac4a0a2423e95328deb5807f787 [term 1 LEADER]: Becoming Leader. State: Replica: 78a6aac4a0a2423e95328deb5807f787, State: Running, Role: LEADER
I20260928 21:54:20.617946 7426 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78a6aac4a0a2423e95328deb5807f787 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:54:20.617930 7431 consensus_queue.cc:237] T 00000000000000000000000000000000 P 78a6aac4a0a2423e95328deb5807f787 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78a6aac4a0a2423e95328deb5807f787" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43003 } }
I20260928 21:54:20.618446 7433 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78a6aac4a0a2423e95328deb5807f787 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "78a6aac4a0a2423e95328deb5807f787" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78a6aac4a0a2423e95328deb5807f787" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43003 } } }
I20260928 21:54:20.618458 7432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78a6aac4a0a2423e95328deb5807f787 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78a6aac4a0a2423e95328deb5807f787. Latest consensus state: current_term: 1 leader_uuid: "78a6aac4a0a2423e95328deb5807f787" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78a6aac4a0a2423e95328deb5807f787" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43003 } } }
I20260928 21:54:20.618656 7432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78a6aac4a0a2423e95328deb5807f787 [sys.catalog]: This master's current role is: LEADER
I20260928 21:54:20.618640 7433 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78a6aac4a0a2423e95328deb5807f787 [sys.catalog]: This master's current role is: LEADER
I20260928 21:54:20.619038 7440 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:54:20.619545 7440 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:54:20.621074 7440 catalog_manager.cc:1347] Generated new cluster ID: f85741d7b6f24ee2815db0111e10abff
I20260928 21:54:20.621132 7440 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:54:20.636457 7440 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:54:20.636659 7440 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:54:20.648998 7440 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 78a6aac4a0a2423e95328deb5807f787: Generated new TSK 0
I20260928 21:54:20.649310 7440 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:54:20.651639 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632440055392-2777-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632440055392-2777-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632440055392-2777-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.65:0
--local_ip_for_outbound_sockets=127.2.182.65
--webserver_interface=127.2.182.65
--webserver_port=0
--tserver_master_addrs=127.2.182.126:43003
--builtin_ntp_servers=127.2.182.84:34099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 21:54:20.745500 7450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:54:20.745729 7450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:54:20.745760 7450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:54:20.747067 7450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:54:20.747121 7450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:54:20.747156 7450 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.65
I20260928 21:54:20.748631 7450 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:34099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632440055392-2777-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632440055392-2777-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.182.65
--webserver_port=0
--tserver_master_addrs=127.2.182.126:43003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7450
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.65
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632440055392-2777-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:54:20.748888 7450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:54:20.749171 7450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:54:20.755088 7455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:20.755119 7458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:20.755132 7456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:20.755465 7450 server_base.cc:1034] running on GCE node
I20260928 21:54:20.755722 7450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:20.755991 7450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:20.757203 7450 hybrid_clock.cc:648] HybridClock initialized: now 1790632460757170 us; error 39 us; skew 500 ppm
I20260928 21:54:20.759052 7450 webserver.cc:460] Webserver started at http://127.2.182.65:40211/ using document root <none> and password file <none>
I20260928 21:54:20.759282 7450 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:20.759352 7450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:20.759488 7450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:54:20.760643 7450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632440055392-2777-0/minicluster-data/ts-0/data/instance:
uuid: "d90d1edf661c459d87c7c8da3337dab0"
format_stamp: "Formatted at 2026-09-28 21:54:20 on dist-test-slave-k6lz"
server_key: "b570c8bad16e9fe984b5a168c4b97ed0"
server_key_iv: "1dc4cc8ca0b7deced4ed964c9be1ea8b"
server_key_version: "encryptionkey@0"
I20260928 21:54:20.760994 7450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632440055392-2777-0/minicluster-data/ts-0/wal/instance:
uuid: "d90d1edf661c459d87c7c8da3337dab0"
format_stamp: "Formatted at 2026-09-28 21:54:20 on dist-test-slave-k6lz"
server_key: "b570c8bad16e9fe984b5a168c4b97ed0"
server_key_iv: "1dc4cc8ca0b7deced4ed964c9be1ea8b"
server_key_version: "encryptionkey@0"
I20260928 21:54:20.762270 7450 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 21:54:20.763010 7464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:20.763204 7450 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:54:20.763300 7450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632440055392-2777-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632440055392-2777-0/minicluster-data/ts-0/wal
uuid: "d90d1edf661c459d87c7c8da3337dab0"
format_stamp: "Formatted at 2026-09-28 21:54:20 on dist-test-slave-k6lz"
server_key: "b570c8bad16e9fe984b5a168c4b97ed0"
server_key_iv: "1dc4cc8ca0b7deced4ed964c9be1ea8b"
server_key_version: "encryptionkey@0"
I20260928 21:54:20.763407 7450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632440055392-2777-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632440055392-2777-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632440055392-2777-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:20.766867 7450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:54:20.767133 7450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:54:20.767259 7450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:54:20.767490 7450 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:54:20.767799 7450 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:54:20.767832 7450 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:20.767858 7450 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:54:20.767876 7450 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:20.773408 7450 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.65:34771
I20260928 21:54:20.773461 7577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.65:34771 every 8 connection(s)
I20260928 21:54:20.773805 7450 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
I20260928 21:54:20.776896 7430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:20.774560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.65:49703 (local address 127.2.182.126:43003)
0928 21:54:20.774604 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:20.774605 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:20.774848 (+ 243us) server_negotiation.cc:407] Connection header received
0928 21:54:20.775396 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:20.775398 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:20.775408 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:20.775466 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:20.776645 (+ 1179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:20.776647 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:20.776647 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:54:20.776652 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:20.776671 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:54:20.776675 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:20.776723 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:20.776805 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 21:54:20.776823 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 21:54:20.777100 7580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:20.774555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:43003 (local address 127.2.182.65:49703)
0928 21:54:20.774723 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:20.774759 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:20.775269 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:20.775539 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:20.775543 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:20.775554 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:20.776467 (+ 913us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:54:20.776468 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:20.776498 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:54:20.776529 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:20.776730 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:20.776736 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:20.776740 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:54:20.776811 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 21:54:20.776886 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":71,"threads_started":1}
I20260928 21:54:20.777581 7578 heartbeater.cc:347] Connected to a master server at 127.2.182.126:43003
I20260928 21:54:20.777670 7578 heartbeater.cc:464] Registering TS with master...
I20260928 21:54:20.777789 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 7450
I20260928 21:54:20.777876 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632440055392-2777-0/minicluster-data/ts-0/wal/instance
I20260928 21:54:20.777806 7578 heartbeater.cc:511] Master 127.2.182.126:43003 requested a full tablet report, sending...
I20260928 21:54:20.778059 2777 external_mini_cluster.cc:1428] Setting key 9f5ae290fb44b5c3ae9f8b42ee9354fa
I20260928 21:54:20.778496 7390 ts_manager.cc:194] Registered new tserver with Master: d90d1edf661c459d87c7c8da3337dab0 (127.2.182.65:34771)
I20260928 21:54:20.779134 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632440055392-2777-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632440055392-2777-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632440055392-2777-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.66:0
--local_ip_for_outbound_sockets=127.2.182.66
--webserver_interface=127.2.182.66
--webserver_port=0
--tserver_master_addrs=127.2.182.126:43003
--builtin_ntp_servers=127.2.182.84:34099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 21:54:20.860193 7581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:54:20.860433 7581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:54:20.860463 7581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:54:20.861709 7581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:54:20.861755 7581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:54:20.861789 7581 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.66
I20260928 21:54:20.863173 7581 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:34099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632440055392-2777-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632440055392-2777-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.182.66
--webserver_port=0
--tserver_master_addrs=127.2.182.126:43003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.66
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632440055392-2777-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:54:20.863459 7581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:54:20.863747 7581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:54:20.868952 7581 server_base.cc:1034] running on GCE node
W20260928 21:54:20.868927 7587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:20.868934 7589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:20.868934 7586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:20.869441 7581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:20.869692 7581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:20.870874 7581 hybrid_clock.cc:648] HybridClock initialized: now 1790632460870855 us; error 36 us; skew 500 ppm
I20260928 21:54:20.872290 7581 webserver.cc:460] Webserver started at http://127.2.182.66:43921/ using document root <none> and password file <none>
I20260928 21:54:20.872619 7581 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:20.872687 7581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:20.872815 7581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:54:20.873914 7581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632440055392-2777-0/minicluster-data/ts-1/data/instance:
uuid: "5f4f64b4fdb54e4384d5e462c6bdcbe0"
format_stamp: "Formatted at 2026-09-28 21:54:20 on dist-test-slave-k6lz"
server_key: "05d69d98fec8fc3652a0b3e0ce7fc8a8"
server_key_iv: "3e8f7de8ec791791e4438ad3451811a5"
server_key_version: "encryptionkey@0"
I20260928 21:54:20.874266 7581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632440055392-2777-0/minicluster-data/ts-1/wal/instance:
uuid: "5f4f64b4fdb54e4384d5e462c6bdcbe0"
format_stamp: "Formatted at 2026-09-28 21:54:20 on dist-test-slave-k6lz"
server_key: "05d69d98fec8fc3652a0b3e0ce7fc8a8"
server_key_iv: "3e8f7de8ec791791e4438ad3451811a5"
server_key_version: "encryptionkey@0"
I20260928 21:54:20.875566 7581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 21:54:20.876405 7595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:20.876574 7581 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:54:20.876653 7581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632440055392-2777-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632440055392-2777-0/minicluster-data/ts-1/wal
uuid: "5f4f64b4fdb54e4384d5e462c6bdcbe0"
format_stamp: "Formatted at 2026-09-28 21:54:20 on dist-test-slave-k6lz"
server_key: "05d69d98fec8fc3652a0b3e0ce7fc8a8"
server_key_iv: "3e8f7de8ec791791e4438ad3451811a5"
server_key_version: "encryptionkey@0"
I20260928 21:54:20.876767 7581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632440055392-2777-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632440055392-2777-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632440055392-2777-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:20.880231 7581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:54:20.880488 7581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:54:20.880605 7581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:54:20.880800 7581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:54:20.881090 7581 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:54:20.881121 7581 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:20.881148 7581 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:54:20.881172 7581 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:20.887006 7581 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.66:37861
I20260928 21:54:20.887044 7708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.66:37861 every 8 connection(s)
I20260928 21:54:20.887424 7581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
I20260928 21:54:20.890477 7430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:20.888161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.66:52571 (local address 127.2.182.126:43003)
0928 21:54:20.888221 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:20.888223 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:20.888499 (+ 276us) server_negotiation.cc:407] Connection header received
0928 21:54:20.889084 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:20.889085 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:20.889091 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:20.889143 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:20.890277 (+ 1134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:20.890278 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:20.890279 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:54:20.890283 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:20.890298 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:54:20.890302 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:20.890359 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:20.890411 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 21:54:20.890423 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 21:54:20.890658 7711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:20.888148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:43003 (local address 127.2.182.66:52571)
0928 21:54:20.888378 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:20.888416 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:20.888938 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:20.889210 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:20.889214 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:20.889222 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:20.890138 (+ 916us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:54:20.890139 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:20.890168 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:54:20.890197 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:20.890359 (+ 162us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:20.890361 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:20.890363 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 21:54:20.890412 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 21:54:20.890471 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":91,"threads_started":1}
I20260928 21:54:20.891068 7709 heartbeater.cc:347] Connected to a master server at 127.2.182.126:43003
I20260928 21:54:20.891160 7709 heartbeater.cc:464] Registering TS with master...
I20260928 21:54:20.891270 7709 heartbeater.cc:511] Master 127.2.182.126:43003 requested a full tablet report, sending...
I20260928 21:54:20.891646 7390 ts_manager.cc:194] Registered new tserver with Master: 5f4f64b4fdb54e4384d5e462c6bdcbe0 (127.2.182.66:37861)
I20260928 21:54:20.893865 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 7581
I20260928 21:54:20.893947 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632440055392-2777-0/minicluster-data/ts-1/wal/instance
I20260928 21:54:20.894076 2777 external_mini_cluster.cc:1428] Setting key 2ffcb7b2d4e2d61c788a99cae455e282
I20260928 21:54:20.894917 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632440055392-2777-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632440055392-2777-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632440055392-2777-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.67:0
--local_ip_for_outbound_sockets=127.2.182.67
--webserver_interface=127.2.182.67
--webserver_port=0
--tserver_master_addrs=127.2.182.126:43003
--builtin_ntp_servers=127.2.182.84:34099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 21:54:20.976478 7712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:54:20.976724 7712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:54:20.976766 7712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:54:20.978061 7712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:54:20.978109 7712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:54:20.978147 7712 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.67
I20260928 21:54:20.979681 7712 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:34099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632440055392-2777-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632440055392-2777-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.182.67
--webserver_port=0
--tserver_master_addrs=127.2.182.126:43003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7712
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.67
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632440055392-2777-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:54:20.979948 7712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:54:20.980369 7712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:54:20.985533 7718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:20.985548 7720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:20.985534 7717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:20.985625 7712 server_base.cc:1034] running on GCE node
I20260928 21:54:20.985970 7712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:20.986189 7712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:20.987388 7712 hybrid_clock.cc:648] HybridClock initialized: now 1790632460987370 us; error 29 us; skew 500 ppm
I20260928 21:54:20.988771 7712 webserver.cc:460] Webserver started at http://127.2.182.67:43725/ using document root <none> and password file <none>
I20260928 21:54:20.989041 7712 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:20.989096 7712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:20.989209 7712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:54:20.990427 7712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632440055392-2777-0/minicluster-data/ts-2/data/instance:
uuid: "3e8a2fd924f64f22accfe653ccdc8a99"
format_stamp: "Formatted at 2026-09-28 21:54:20 on dist-test-slave-k6lz"
server_key: "5247ed9ae98eab6231553984d20c13ca"
server_key_iv: "6f5e603cae304cad80e7e20f651738bc"
server_key_version: "encryptionkey@0"
I20260928 21:54:20.990753 7712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632440055392-2777-0/minicluster-data/ts-2/wal/instance:
uuid: "3e8a2fd924f64f22accfe653ccdc8a99"
format_stamp: "Formatted at 2026-09-28 21:54:20 on dist-test-slave-k6lz"
server_key: "5247ed9ae98eab6231553984d20c13ca"
server_key_iv: "6f5e603cae304cad80e7e20f651738bc"
server_key_version: "encryptionkey@0"
I20260928 21:54:20.992195 7712 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 21:54:20.993060 7726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:20.993283 7712 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:54:20.993389 7712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632440055392-2777-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632440055392-2777-0/minicluster-data/ts-2/wal
uuid: "3e8a2fd924f64f22accfe653ccdc8a99"
format_stamp: "Formatted at 2026-09-28 21:54:20 on dist-test-slave-k6lz"
server_key: "5247ed9ae98eab6231553984d20c13ca"
server_key_iv: "6f5e603cae304cad80e7e20f651738bc"
server_key_version: "encryptionkey@0"
I20260928 21:54:20.993462 7712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632440055392-2777-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632440055392-2777-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632440055392-2777-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:20.997061 7712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:54:20.997319 7712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:54:20.997416 7712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:54:20.997570 7712 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:54:20.997849 7712 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:54:20.997881 7712 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:20.997901 7712 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:54:20.997912 7712 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:21.003834 7712 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.67:42565
I20260928 21:54:21.003893 7839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.67:42565 every 8 connection(s)
I20260928 21:54:21.004222 7712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
I20260928 21:54:21.008095 7430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:21.004970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.67:55269 (local address 127.2.182.126:43003)
0928 21:54:21.005226 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:21.005229 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:21.005248 (+ 19us) server_negotiation.cc:407] Connection header received
0928 21:54:21.005738 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:21.005740 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:21.005745 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:21.005904 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:21.007674 (+ 1770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:21.007675 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:21.007675 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:54:21.007680 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:21.007698 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:54:21.007702 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:21.007759 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:21.008016 (+ 257us) server_negotiation.cc:299] Negotiation successful
0928 21:54:21.008037 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260928 21:54:21.008216 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:21.004958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:43003 (local address 127.2.182.67:55269)
0928 21:54:21.005136 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:21.005179 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:21.005632 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:21.006026 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:21.006030 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:21.006039 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:21.007481 (+ 1442us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:54:21.007482 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:21.007518 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:54:21.007554 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:21.007867 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:21.007873 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:21.007876 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:54:21.007967 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 21:54:21.008035 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":78,"threads_started":1}
I20260928 21:54:21.008962 7840 heartbeater.cc:347] Connected to a master server at 127.2.182.126:43003
I20260928 21:54:21.009055 7840 heartbeater.cc:464] Registering TS with master...
I20260928 21:54:21.009171 7840 heartbeater.cc:511] Master 127.2.182.126:43003 requested a full tablet report, sending...
I20260928 21:54:21.009523 7390 ts_manager.cc:194] Registered new tserver with Master: 3e8a2fd924f64f22accfe653ccdc8a99 (127.2.182.67:42565)
I20260928 21:54:21.009655 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 7712
I20260928 21:54:21.009752 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632440055392-2777-0/minicluster-data/ts-2/wal/instance
I20260928 21:54:21.009928 2777 external_mini_cluster.cc:1428] Setting key 786dc7b0c3a481481b7f13aef82639e0
I20260928 21:54:21.010856 2777 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:54:21.013460 7430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:21.012246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33996 (local address 127.2.182.126:43003)
0928 21:54:21.012327 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:21.012330 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:21.012487 (+ 157us) server_negotiation.cc:407] Connection header received
0928 21:54:21.012536 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:21.012537 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:21.012542 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:21.012588 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:21.013246 (+ 658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:21.013247 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:21.013247 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:54:21.013251 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:21.013269 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:54:21.013274 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:21.013333 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:21.013374 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 21:54:21.013387 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 21:54:21.015822 7390 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33996:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:54:21.017568 7390 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:54:21.022531 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:21.020404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.66:37861 (local address 127.0.0.1:39152)
0928 21:54:21.020852 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:21.020896 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:21.021003 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:21.021461 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:21.021462 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:21.021465 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:21.022114 (+ 649us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:54:21.022115 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:21.022130 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:54:21.022151 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:21.022366 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:21.022372 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:21.022375 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:54:21.022435 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 21:54:21.022449 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":73,"threads_started":1}
I20260928 21:54:21.022541 7851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:21.020492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39152 (local address 127.2.182.66:37861)
0928 21:54:21.020645 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:21.020650 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:21.020972 (+ 322us) server_negotiation.cc:407] Connection header received
0928 21:54:21.021062 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:21.021067 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:21.021078 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:21.021141 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:21.022242 (+ 1101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:21.022251 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:21.022251 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:54:21.022255 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:21.022290 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:54:21.022313 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:21.022363 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:21.022447 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 21:54:21.022461 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":88,"threads_started":1}
I20260928 21:54:21.022667 7850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:21.020406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.67:42565 (local address 127.0.0.1:54642)
0928 21:54:21.021154 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:21.021163 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:21.021212 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:21.021425 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:21.021431 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:21.021438 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:21.022325 (+ 887us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:54:21.022326 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:21.022331 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:54:21.022340 (+ 9us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:21.022558 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:21.022559 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:21.022559 (+ 0us) client_negotiation.cc:769] Sending connection context
0928 21:54:21.022597 (+ 38us) client_negotiation.cc:240] Negotiation successful
0928 21:54:21.022609 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":731,"spinlock_wait_cycles":896,"thread_start_us":158,"threads_started":1}
I20260928 21:54:21.022661 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:21.020709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45412 (local address 127.2.182.65:34771)
0928 21:54:21.020867 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:21.020870 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:21.021348 (+ 478us) server_negotiation.cc:407] Connection header received
0928 21:54:21.021458 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:21.021464 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:21.021474 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:21.021537 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:21.022316 (+ 779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:21.022325 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:21.022325 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:54:21.022330 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:21.022365 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:54:21.022406 (+ 41us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:21.022442 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:21.022557 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 21:54:21.022574 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":79,"threads_started":1}
I20260928 21:54:21.022861 7853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:21.020710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54642 (local address 127.2.182.67:42565)
0928 21:54:21.020990 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:21.020994 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:21.021209 (+ 215us) server_negotiation.cc:407] Connection header received
0928 21:54:21.021328 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:21.021332 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:21.021341 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:21.021411 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:21.022418 (+ 1007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:21.022425 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:21.022426 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:54:21.022430 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:21.022464 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:54:21.022491 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:21.022537 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:21.022769 (+ 232us) server_negotiation.cc:299] Negotiation successful
0928 21:54:21.022784 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":197,"threads_started":1}
I20260928 21:54:21.022907 7849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:21.020407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.65:34771 (local address 127.0.0.1:45412)
0928 21:54:21.021290 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:21.021298 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:21.021378 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:21.021894 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:21.021895 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:21.021898 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:21.022203 (+ 305us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:54:21.022203 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:21.022212 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:54:21.022225 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:21.022484 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:21.022484 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:21.022485 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 21:54:21.022842 (+ 357us) client_negotiation.cc:240] Negotiation successful
0928 21:54:21.022854 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":870,"thread_start_us":88,"threads_started":1}
I20260928 21:54:21.023875 7512 tablet_service.cc:1467] Processing CreateTablet for tablet d33801287088470c954e06ca1fb8d759 (DEFAULT_TABLE table=test-table [id=a8456f17e46d43128c571212f015a2d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:54:21.023921 7774 tablet_service.cc:1467] Processing CreateTablet for tablet d33801287088470c954e06ca1fb8d759 (DEFAULT_TABLE table=test-table [id=a8456f17e46d43128c571212f015a2d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:54:21.024051 7643 tablet_service.cc:1467] Processing CreateTablet for tablet d33801287088470c954e06ca1fb8d759 (DEFAULT_TABLE table=test-table [id=a8456f17e46d43128c571212f015a2d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:54:21.024281 7774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d33801287088470c954e06ca1fb8d759. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:54:21.024430 7643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d33801287088470c954e06ca1fb8d759. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:54:21.024473 7512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d33801287088470c954e06ca1fb8d759. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:54:21.027496 7855 tablet_bootstrap.cc:492] T d33801287088470c954e06ca1fb8d759 P 3e8a2fd924f64f22accfe653ccdc8a99: Bootstrap starting.
I20260928 21:54:21.027558 7854 tablet_bootstrap.cc:492] T d33801287088470c954e06ca1fb8d759 P d90d1edf661c459d87c7c8da3337dab0: Bootstrap starting.
I20260928 21:54:21.028239 7855 tablet_bootstrap.cc:654] T d33801287088470c954e06ca1fb8d759 P 3e8a2fd924f64f22accfe653ccdc8a99: Neither blocks nor log segments found. Creating new log.
I20260928 21:54:21.028250 7854 tablet_bootstrap.cc:654] T d33801287088470c954e06ca1fb8d759 P d90d1edf661c459d87c7c8da3337dab0: Neither blocks nor log segments found. Creating new log.
I20260928 21:54:21.028568 7855 log.cc:824] T d33801287088470c954e06ca1fb8d759 P 3e8a2fd924f64f22accfe653ccdc8a99: Log is configured to *not* fsync() on all Append() calls
I20260928 21:54:21.028671 7854 log.cc:824] T d33801287088470c954e06ca1fb8d759 P d90d1edf661c459d87c7c8da3337dab0: Log is configured to *not* fsync() on all Append() calls
I20260928 21:54:21.028995 7856 tablet_bootstrap.cc:492] T d33801287088470c954e06ca1fb8d759 P 5f4f64b4fdb54e4384d5e462c6bdcbe0: Bootstrap starting.
I20260928 21:54:21.029361 7854 tablet_bootstrap.cc:492] T d33801287088470c954e06ca1fb8d759 P d90d1edf661c459d87c7c8da3337dab0: No bootstrap required, opened a new log
I20260928 21:54:21.029419 7854 ts_tablet_manager.cc:1397] T d33801287088470c954e06ca1fb8d759 P d90d1edf661c459d87c7c8da3337dab0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:54:21.031268 7854 raft_consensus.cc:348] T d33801287088470c954e06ca1fb8d759 P d90d1edf661c459d87c7c8da3337dab0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e8a2fd924f64f22accfe653ccdc8a99" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 42565 } } peers { permanent_uuid: "d90d1edf661c459d87c7c8da3337dab0" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 34771 } } peers { permanent_uuid: "5f4f64b4fdb54e4384d5e462c6bdcbe0" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37861 } }
I20260928 21:54:21.031476 7854 raft_consensus.cc:374] T d33801287088470c954e06ca1fb8d759 P d90d1edf661c459d87c7c8da3337dab0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:54:21.031514 7854 raft_consensus.cc:729] T d33801287088470c954e06ca1fb8d759 P d90d1edf661c459d87c7c8da3337dab0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d90d1edf661c459d87c7c8da3337dab0, State: Initialized, Role: FOLLOWER
I20260928 21:54:21.031635 7854 consensus_queue.cc:260] T d33801287088470c954e06ca1fb8d759 P d90d1edf661c459d87c7c8da3337dab0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e8a2fd924f64f22accfe653ccdc8a99" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 42565 } } peers { permanent_uuid: "d90d1edf661c459d87c7c8da3337dab0" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 34771 } } peers { permanent_uuid: "5f4f64b4fdb54e4384d5e462c6bdcbe0" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37861 } }
I20260928 21:54:21.031924 7854 ts_tablet_manager.cc:1428] T d33801287088470c954e06ca1fb8d759 P d90d1edf661c459d87c7c8da3337dab0: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 21:54:21.031973 7578 heartbeater.cc:503] Master 127.2.182.126:43003 was elected leader, sending a full tablet report...
I20260928 21:54:21.032051 7856 tablet_bootstrap.cc:654] T d33801287088470c954e06ca1fb8d759 P 5f4f64b4fdb54e4384d5e462c6bdcbe0: Neither blocks nor log segments found. Creating new log.
I20260928 21:54:21.032356 7856 log.cc:824] T d33801287088470c954e06ca1fb8d759 P 5f4f64b4fdb54e4384d5e462c6bdcbe0: Log is configured to *not* fsync() on all Append() calls
I20260928 21:54:21.033123 7855 tablet_bootstrap.cc:492] T d33801287088470c954e06ca1fb8d759 P 3e8a2fd924f64f22accfe653ccdc8a99: No bootstrap required, opened a new log
I20260928 21:54:21.033121 7856 tablet_bootstrap.cc:492] T d33801287088470c954e06ca1fb8d759 P 5f4f64b4fdb54e4384d5e462c6bdcbe0: No bootstrap required, opened a new log
I20260928 21:54:21.033197 7856 ts_tablet_manager.cc:1397] T d33801287088470c954e06ca1fb8d759 P 5f4f64b4fdb54e4384d5e462c6bdcbe0: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260928 21:54:21.033197 7855 ts_tablet_manager.cc:1397] T d33801287088470c954e06ca1fb8d759 P 3e8a2fd924f64f22accfe653ccdc8a99: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260928 21:54:21.036023 7855 raft_consensus.cc:348] T d33801287088470c954e06ca1fb8d759 P 3e8a2fd924f64f22accfe653ccdc8a99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e8a2fd924f64f22accfe653ccdc8a99" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 42565 } } peers { permanent_uuid: "d90d1edf661c459d87c7c8da3337dab0" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 34771 } } peers { permanent_uuid: "5f4f64b4fdb54e4384d5e462c6bdcbe0" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37861 } }
I20260928 21:54:21.036023 7856 raft_consensus.cc:348] T d33801287088470c954e06ca1fb8d759 P 5f4f64b4fdb54e4384d5e462c6bdcbe0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e8a2fd924f64f22accfe653ccdc8a99" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 42565 } } peers { permanent_uuid: "d90d1edf661c459d87c7c8da3337dab0" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 34771 } } peers { permanent_uuid: "5f4f64b4fdb54e4384d5e462c6bdcbe0" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37861 } }
I20260928 21:54:21.036224 7855 raft_consensus.cc:374] T d33801287088470c954e06ca1fb8d759 P 3e8a2fd924f64f22accfe653ccdc8a99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:54:21.036224 7856 raft_consensus.cc:374] T d33801287088470c954e06ca1fb8d759 P 5f4f64b4fdb54e4384d5e462c6bdcbe0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:54:21.036259 7855 raft_consensus.cc:729] T d33801287088470c954e06ca1fb8d759 P 3e8a2fd924f64f22accfe653ccdc8a99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e8a2fd924f64f22accfe653ccdc8a99, State: Initialized, Role: FOLLOWER
I20260928 21:54:21.036259 7856 raft_consensus.cc:729] T d33801287088470c954e06ca1fb8d759 P 5f4f64b4fdb54e4384d5e462c6bdcbe0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f4f64b4fdb54e4384d5e462c6bdcbe0, State: Initialized, Role: FOLLOWER
I20260928 21:54:21.036386 7856 consensus_queue.cc:260] T d33801287088470c954e06ca1fb8d759 P 5f4f64b4fdb54e4384d5e462c6bdcbe0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e8a2fd924f64f22accfe653ccdc8a99" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 42565 } } peers { permanent_uuid: "d90d1edf661c459d87c7c8da3337dab0" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 34771 } } peers { permanent_uuid: "5f4f64b4fdb54e4384d5e462c6bdcbe0" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37861 } }
I20260928 21:54:21.036386 7855 consensus_queue.cc:260] T d33801287088470c954e06ca1fb8d759 P 3e8a2fd924f64f22accfe653ccdc8a99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e8a2fd924f64f22accfe653ccdc8a99" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 42565 } } peers { permanent_uuid: "d90d1edf661c459d87c7c8da3337dab0" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 34771 } } peers { permanent_uuid: "5f4f64b4fdb54e4384d5e462c6bdcbe0" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37861 } }
I20260928 21:54:21.036670 7856 ts_tablet_manager.cc:1428] T d33801287088470c954e06ca1fb8d759 P 5f4f64b4fdb54e4384d5e462c6bdcbe0: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260928 21:54:21.036670 7855 ts_tablet_manager.cc:1428] T d33801287088470c954e06ca1fb8d759 P 3e8a2fd924f64f22accfe653ccdc8a99: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260928 21:54:21.036798 7709 heartbeater.cc:503] Master 127.2.182.126:43003 was elected leader, sending a full tablet report...
I20260928 21:54:21.036695 7840 heartbeater.cc:503] Master 127.2.182.126:43003 was elected leader, sending a full tablet report...
I20260928 21:54:21.042840 7859 raft_consensus.cc:482] T d33801287088470c954e06ca1fb8d759 P d90d1edf661c459d87c7c8da3337dab0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:54:21.043043 7859 raft_consensus.cc:504] T d33801287088470c954e06ca1fb8d759 P d90d1edf661c459d87c7c8da3337dab0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e8a2fd924f64f22accfe653ccdc8a99" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 42565 } } peers { permanent_uuid: "d90d1edf661c459d87c7c8da3337dab0" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 34771 } } peers { permanent_uuid: "5f4f64b4fdb54e4384d5e462c6bdcbe0" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37861 } }
I20260928 21:54:21.043442 7859 leader_election.cc:288] T d33801287088470c954e06ca1fb8d759 P d90d1edf661c459d87c7c8da3337dab0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e8a2fd924f64f22accfe653ccdc8a99 (127.2.182.67:42565), 5f4f64b4fdb54e4384d5e462c6bdcbe0 (127.2.182.66:37861)
I20260928 21:54:21.045042 7851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:21.043668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.65:58751 (local address 127.2.182.66:37861)
0928 21:54:21.043725 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:21.043727 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:21.044104 (+ 377us) server_negotiation.cc:407] Connection header received
0928 21:54:21.044132 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:21.044133 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:21.044139 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:21.044190 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:21.044835 (+ 645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:21.044836 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:21.044837 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:54:21.044841 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:21.044859 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:54:21.044864 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:21.044919 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:21.044960 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 21:54:21.044972 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 21:54:21.045138 7853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:21.043769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.65:57219 (local address 127.2.182.67:42565)
0928 21:54:21.043883 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:21.043885 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:21.043894 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:54:21.043913 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:21.043914 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:21.043918 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:21.043966 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:21.044918 (+ 952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:21.044920 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:21.044920 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:54:21.044926 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:21.044943 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:54:21.044948 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:21.044999 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:21.045067 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 21:54:21.045079 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260928 21:54:21.045064 7863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:21.043612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.66:37861 (local address 127.2.182.65:58751)
0928 21:54:21.044041 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:21.044049 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:21.044088 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:21.044196 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:21.044196 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:21.044208 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:21.044718 (+ 510us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:54:21.044719 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:21.044727 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:54:21.044739 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:21.044917 (+ 178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:21.044918 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:21.044918 (+ 0us) client_negotiation.cc:769] Sending connection context
0928 21:54:21.044962 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 21:54:21.044976 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"mutex_wait_us":40,"thread_start_us":108,"threads_started":1}
I20260928 21:54:21.045150 7580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:21.043612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.67:42565 (local address 127.2.182.65:57219)
0928 21:54:21.043663 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:21.043676 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:21.043724 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:21.043986 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:21.043986 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:21.043989 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:21.044732 (+ 743us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:54:21.044733 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:21.044739 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:54:21.044751 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:21.045017 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:21.045018 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:21.045019 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 21:54:21.045064 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 21:54:21.045076 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 21:54:21.045522 7663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d33801287088470c954e06ca1fb8d759" candidate_uuid: "d90d1edf661c459d87c7c8da3337dab0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f4f64b4fdb54e4384d5e462c6bdcbe0" is_pre_election: true
I20260928 21:54:21.045631 7794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d33801287088470c954e06ca1fb8d759" candidate_uuid: "d90d1edf661c459d87c7c8da3337dab0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e8a2fd924f64f22accfe653ccdc8a99" is_pre_election: true
I20260928 21:54:21.045715 7663 raft_consensus.cc:2454] T d33801287088470c954e06ca1fb8d759 P 5f4f64b4fdb54e4384d5e462c6bdcbe0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d90d1edf661c459d87c7c8da3337dab0 in term 0.
I20260928 21:54:21.045763 7794 raft_consensus.cc:2454] T d33801287088470c954e06ca1fb8d759 P 3e8a2fd924f64f22accfe653ccdc8a99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d90d1edf661c459d87c7c8da3337dab0 in term 0.
I20260928 21:54:21.046007 7466 leader_election.cc:302] T d33801287088470c954e06ca1fb8d759 P d90d1edf661c459d87c7c8da3337dab0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f4f64b4fdb54e4384d5e462c6bdcbe0, d90d1edf661c459d87c7c8da3337dab0; no voters:
I20260928 21:54:21.046151 7859 raft_consensus.cc:2781] T d33801287088470c954e06ca1fb8d759 P d90d1edf661c459d87c7c8da3337dab0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:54:21.046204 7859 raft_consensus.cc:482] T d33801287088470c954e06ca1fb8d759 P d90d1edf661c459d87c7c8da3337dab0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:54:21.046221 7859 raft_consensus.cc:3037] T d33801287088470c954e06ca1fb8d759 P d90d1edf661c459d87c7c8da3337dab0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:54:21.047001 7859 raft_consensus.cc:504] T d33801287088470c954e06ca1fb8d759 P d90d1edf661c459d87c7c8da3337dab0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e8a2fd924f64f22accfe653ccdc8a99" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 42565 } } peers { permanent_uuid: "d90d1edf661c459d87c7c8da3337dab0" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 34771 } } peers { permanent_uuid: "5f4f64b4fdb54e4384d5e462c6bdcbe0" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37861 } }
I20260928 21:54:21.047127 7859 leader_election.cc:288] T d33801287088470c954e06ca1fb8d759 P d90d1edf661c459d87c7c8da3337dab0 [CANDIDATE]: Term 1 election: Requested vote from peers 3e8a2fd924f64f22accfe653ccdc8a99 (127.2.182.67:42565), 5f4f64b4fdb54e4384d5e462c6bdcbe0 (127.2.182.66:37861)
I20260928 21:54:21.047282 7794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d33801287088470c954e06ca1fb8d759" candidate_uuid: "d90d1edf661c459d87c7c8da3337dab0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e8a2fd924f64f22accfe653ccdc8a99"
I20260928 21:54:21.047382 7794 raft_consensus.cc:3037] T d33801287088470c954e06ca1fb8d759 P 3e8a2fd924f64f22accfe653ccdc8a99 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:54:21.047302 7663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d33801287088470c954e06ca1fb8d759" candidate_uuid: "d90d1edf661c459d87c7c8da3337dab0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f4f64b4fdb54e4384d5e462c6bdcbe0"
I20260928 21:54:21.047470 7663 raft_consensus.cc:3037] T d33801287088470c954e06ca1fb8d759 P 5f4f64b4fdb54e4384d5e462c6bdcbe0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:54:21.048180 7794 raft_consensus.cc:2454] T d33801287088470c954e06ca1fb8d759 P 3e8a2fd924f64f22accfe653ccdc8a99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d90d1edf661c459d87c7c8da3337dab0 in term 1.
I20260928 21:54:21.048347 7663 raft_consensus.cc:2454] T d33801287088470c954e06ca1fb8d759 P 5f4f64b4fdb54e4384d5e462c6bdcbe0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d90d1edf661c459d87c7c8da3337dab0 in term 1.
I20260928 21:54:21.048372 7468 leader_election.cc:302] T d33801287088470c954e06ca1fb8d759 P d90d1edf661c459d87c7c8da3337dab0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e8a2fd924f64f22accfe653ccdc8a99, d90d1edf661c459d87c7c8da3337dab0; no voters:
I20260928 21:54:21.048460 7859 raft_consensus.cc:2781] T d33801287088470c954e06ca1fb8d759 P d90d1edf661c459d87c7c8da3337dab0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:54:21.048631 7859 raft_consensus.cc:686] T d33801287088470c954e06ca1fb8d759 P d90d1edf661c459d87c7c8da3337dab0 [term 1 LEADER]: Becoming Leader. State: Replica: d90d1edf661c459d87c7c8da3337dab0, State: Running, Role: LEADER
I20260928 21:54:21.048709 7859 consensus_queue.cc:237] T d33801287088470c954e06ca1fb8d759 P d90d1edf661c459d87c7c8da3337dab0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e8a2fd924f64f22accfe653ccdc8a99" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 42565 } } peers { permanent_uuid: "d90d1edf661c459d87c7c8da3337dab0" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 34771 } } peers { permanent_uuid: "5f4f64b4fdb54e4384d5e462c6bdcbe0" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37861 } }
I20260928 21:54:21.049567 7389 catalog_manager.cc:5495] T d33801287088470c954e06ca1fb8d759 P d90d1edf661c459d87c7c8da3337dab0 reported cstate change: term changed from 0 to 1, leader changed from <none> to d90d1edf661c459d87c7c8da3337dab0 (127.2.182.65). New cstate: current_term: 1 leader_uuid: "d90d1edf661c459d87c7c8da3337dab0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e8a2fd924f64f22accfe653ccdc8a99" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 42565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d90d1edf661c459d87c7c8da3337dab0" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 34771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f4f64b4fdb54e4384d5e462c6bdcbe0" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37861 } health_report { overall_health: UNKNOWN } } }
I20260928 21:54:21.061223 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:21.059841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45414 (local address 127.2.182.65:34771)
0928 21:54:21.059963 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:21.059966 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:21.059979 (+ 13us) server_negotiation.cc:407] Connection header received
0928 21:54:21.060018 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:21.060019 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:21.060032 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:21.060096 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:21.060848 (+ 752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:21.060850 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:21.060850 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:54:21.060854 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:21.060868 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:54:21.060873 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:21.060917 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:21.061129 (+ 212us) server_negotiation.cc:299] Negotiation successful
0928 21:54:21.061153 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260928 21:54:21.063189 7663 raft_consensus.cc:1261] T d33801287088470c954e06ca1fb8d759 P 5f4f64b4fdb54e4384d5e462c6bdcbe0 [term 1 FOLLOWER]: Refusing update from remote peer d90d1edf661c459d87c7c8da3337dab0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:54:21.063203 7794 raft_consensus.cc:1261] T d33801287088470c954e06ca1fb8d759 P 3e8a2fd924f64f22accfe653ccdc8a99 [term 1 FOLLOWER]: Refusing update from remote peer d90d1edf661c459d87c7c8da3337dab0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:54:21.063591 7859 consensus_queue.cc:1035] T d33801287088470c954e06ca1fb8d759 P d90d1edf661c459d87c7c8da3337dab0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e8a2fd924f64f22accfe653ccdc8a99" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 42565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:54:21.063743 7864 consensus_queue.cc:1035] T d33801287088470c954e06ca1fb8d759 P d90d1edf661c459d87c7c8da3337dab0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f4f64b4fdb54e4384d5e462c6bdcbe0" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:54:21.066324 7867 mvcc.cc:205] Tried to move back new op lower bound from 7334430560512139264 to 7334430560456015872. Current Snapshot: MvccSnapshot[applied={T|T < 7334430560512139264}]
I20260928 21:54:21.067270 7868 mvcc.cc:205] Tried to move back new op lower bound from 7334430560512139264 to 7334430560456015872. Current Snapshot: MvccSnapshot[applied={T|T < 7334430560512139264 or (T in {7334430560512139264})}]
I20260928 21:54:21.067618 7869 mvcc.cc:205] Tried to move back new op lower bound from 7334430560512139264 to 7334430560456015872. Current Snapshot: MvccSnapshot[applied={T|T < 7334430560512139264 or (T in {7334430560512139264})}]
I20260928 21:54:21.070255 7389 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33996:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:54:21.070374 7389 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33996:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:54:21.071413 7389 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 78a6aac4a0a2423e95328deb5807f787: Sending DeleteTablet for 3 replicas of tablet d33801287088470c954e06ca1fb8d759
I20260928 21:54:21.071744 7512 tablet_service.cc:1514] Processing DeleteTablet for tablet d33801287088470c954e06ca1fb8d759 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:54:21 UTC) from {username='slave'} at 127.0.0.1:45412
I20260928 21:54:21.071897 7643 tablet_service.cc:1514] Processing DeleteTablet for tablet d33801287088470c954e06ca1fb8d759 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:54:21 UTC) from {username='slave'} at 127.0.0.1:39152
I20260928 21:54:21.072026 7774 tablet_service.cc:1514] Processing DeleteTablet for tablet d33801287088470c954e06ca1fb8d759 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:54:21 UTC) from {username='slave'} at 127.0.0.1:54642
I20260928 21:54:21.072052 7877 tablet_replica.cc:330] T d33801287088470c954e06ca1fb8d759 P d90d1edf661c459d87c7c8da3337dab0: stopping tablet replica
I20260928 21:54:21.072151 7877 raft_consensus.cc:2229] T d33801287088470c954e06ca1fb8d759 P d90d1edf661c459d87c7c8da3337dab0 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:54:21.072288 7877 raft_consensus.cc:2258] T d33801287088470c954e06ca1fb8d759 P d90d1edf661c459d87c7c8da3337dab0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:54:21.072309 7879 tablet_replica.cc:330] T d33801287088470c954e06ca1fb8d759 P 3e8a2fd924f64f22accfe653ccdc8a99: stopping tablet replica
I20260928 21:54:21.072492 7879 raft_consensus.cc:2229] T d33801287088470c954e06ca1fb8d759 P 3e8a2fd924f64f22accfe653ccdc8a99 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:54:21.072633 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 7450
I20260928 21:54:21.072636 7879 raft_consensus.cc:2258] T d33801287088470c954e06ca1fb8d759 P 3e8a2fd924f64f22accfe653ccdc8a99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:54:21.073518 7879 ts_tablet_manager.cc:1905] T d33801287088470c954e06ca1fb8d759 P 3e8a2fd924f64f22accfe653ccdc8a99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:54:21.076020 7879 ts_tablet_manager.cc:1918] T d33801287088470c954e06ca1fb8d759 P 3e8a2fd924f64f22accfe653ccdc8a99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:54:21.076110 7879 log.cc:1196] T d33801287088470c954e06ca1fb8d759 P 3e8a2fd924f64f22accfe653ccdc8a99: Deleting WAL directory at /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632440055392-2777-0/minicluster-data/ts-2/wal/wals/d33801287088470c954e06ca1fb8d759
I20260928 21:54:21.076457 7879 ts_tablet_manager.cc:1939] T d33801287088470c954e06ca1fb8d759 P 3e8a2fd924f64f22accfe653ccdc8a99: Deleting consensus metadata
I20260928 21:54:21.077394 7377 catalog_manager.cc:4841] TS 3e8a2fd924f64f22accfe653ccdc8a99 (127.2.182.67:42565): tablet d33801287088470c954e06ca1fb8d759 (table test-table [id=a8456f17e46d43128c571212f015a2d0]) successfully deleted
I20260928 21:54:21.079936 7878 tablet_replica.cc:330] T d33801287088470c954e06ca1fb8d759 P 5f4f64b4fdb54e4384d5e462c6bdcbe0: stopping tablet replica
I20260928 21:54:21.080358 7878 raft_consensus.cc:2229] T d33801287088470c954e06ca1fb8d759 P 5f4f64b4fdb54e4384d5e462c6bdcbe0 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260928 21:54:21.080010 7376 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.182.65:34771 (error 108)
I20260928 21:54:21.080554 7878 raft_consensus.cc:2258] T d33801287088470c954e06ca1fb8d759 P 5f4f64b4fdb54e4384d5e462c6bdcbe0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:54:21.080976 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 7581
I20260928 21:54:21.081053 7849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:21.080860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.65:34771 (local address 127.0.0.1:45418)
0928 21:54:21.080967 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:21.081017 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.65:34771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260928 21:54:21.081357 7376 catalog_manager.cc:4584] TS d90d1edf661c459d87c7c8da3337dab0 (127.2.182.65:34771): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d33801287088470c954e06ca1fb8d759: Network error: Client connection negotiation failed: client connection to 127.2.182.65:34771: connect: Connection refused (error 111)
I20260928 21:54:21.087324 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 7712
I20260928 21:54:21.087554 7849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:21.087411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.66:37861 (local address 127.0.0.1:39162)
0928 21:54:21.087475 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:21.087501 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.66:37861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 21:54:21.092036 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 7359
2026-09-28T21:54:21Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (620 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260928 21:54:21.111304 2777 mini_oidc.cc:152] Starting JWKS server
I20260928 21:54:21.112531 2777 webserver.cc:460] Webserver started at https://127.0.0.1:39513/ using document root <none> and password file <none>
I20260928 21:54:21.112624 2777 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 21:54:21.112788 2777 webserver.cc:460] Webserver started at http://127.0.0.1:36375/ using document root <none> and password file <none>
2026-09-28T21:54:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:54:21Z Disabled control of system clock
I20260928 21:54:21.121176 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790632440055392-2777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790632440055392-2777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790632440055392-2777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:43129
--webserver_interface=127.2.182.126
--webserver_port=0
--builtin_ntp_servers=127.2.182.84:43515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.126:43129
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39513/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790632440055392-2777-0/testchainca.cert with env {}
W20260928 21:54:21.207105 7886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:54:21.207360 7886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:54:21.207384 7886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:54:21.208727 7886 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:54:21.208777 7886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:54:21.208788 7886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:54:21.208801 7886 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:54:21.208811 7886 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 21:54:21.208822 7886 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 21:54:21.208833 7886 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39513/jwks.json
I20260928 21:54:21.210515 7886 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:43515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790632440055392-2777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790632440055392-2777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.126:43129
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:43129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39513/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790632440055392-2777-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790632440055392-2777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:54:21.210794 7886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:54:21.211022 7886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:54:21.216408 7892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:21.216408 7894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:21.216408 7891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:21.216722 7886 server_base.cc:1034] running on GCE node
I20260928 21:54:21.216998 7886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:21.217279 7886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:21.218453 7886 hybrid_clock.cc:648] HybridClock initialized: now 1790632461218433 us; error 37 us; skew 500 ppm
I20260928 21:54:21.220124 7886 webserver.cc:460] Webserver started at http://127.2.182.126:34367/ using document root <none> and password file <none>
I20260928 21:54:21.220397 7886 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:21.220451 7886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:21.220569 7886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:54:21.221853 7886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790632440055392-2777-0/minicluster-data/master-0/data/instance:
uuid: "700a574bc5f54e5eb7a32f5d13fcc867"
format_stamp: "Formatted at 2026-09-28 21:54:21 on dist-test-slave-k6lz"
server_key: "d4d75f19d372db514bcc15862b882039"
server_key_iv: "294af013ab386b856ea7db53b3113fa8"
server_key_version: "encryptionkey@0"
I20260928 21:54:21.222232 7886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790632440055392-2777-0/minicluster-data/master-0/wal/instance:
uuid: "700a574bc5f54e5eb7a32f5d13fcc867"
format_stamp: "Formatted at 2026-09-28 21:54:21 on dist-test-slave-k6lz"
server_key: "d4d75f19d372db514bcc15862b882039"
server_key_iv: "294af013ab386b856ea7db53b3113fa8"
server_key_version: "encryptionkey@0"
I20260928 21:54:21.223876 7886 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260928 21:54:21.224886 7900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:21.225219 7886 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:54:21.225385 7886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790632440055392-2777-0/minicluster-data/master-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790632440055392-2777-0/minicluster-data/master-0/wal
uuid: "700a574bc5f54e5eb7a32f5d13fcc867"
format_stamp: "Formatted at 2026-09-28 21:54:21 on dist-test-slave-k6lz"
server_key: "d4d75f19d372db514bcc15862b882039"
server_key_iv: "294af013ab386b856ea7db53b3113fa8"
server_key_version: "encryptionkey@0"
I20260928 21:54:21.225502 7886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790632440055392-2777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790632440055392-2777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790632440055392-2777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:21.236428 7886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:54:21.236755 7886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:54:21.236888 7886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:54:21.240618 7886 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.126:43129
I20260928 21:54:21.240708 7952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.126:43129 every 8 connection(s)
I20260928 21:54:21.241003 7886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
I20260928 21:54:21.242024 7953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:54:21.244378 7953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 700a574bc5f54e5eb7a32f5d13fcc867: Bootstrap starting.
I20260928 21:54:21.244974 7953 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 700a574bc5f54e5eb7a32f5d13fcc867: Neither blocks nor log segments found. Creating new log.
I20260928 21:54:21.245266 7953 log.cc:824] T 00000000000000000000000000000000 P 700a574bc5f54e5eb7a32f5d13fcc867: Log is configured to *not* fsync() on all Append() calls
I20260928 21:54:21.246012 7953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 700a574bc5f54e5eb7a32f5d13fcc867: No bootstrap required, opened a new log
I20260928 21:54:21.246789 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 7886
I20260928 21:54:21.246940 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790632440055392-2777-0/minicluster-data/master-0/wal/instance
I20260928 21:54:21.247116 2777 external_mini_cluster.cc:1428] Setting key fefd7533f958f17b61e63fac01a20a13
I20260928 21:54:21.248292 7953 raft_consensus.cc:348] T 00000000000000000000000000000000 P 700a574bc5f54e5eb7a32f5d13fcc867 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "700a574bc5f54e5eb7a32f5d13fcc867" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43129 } }
I20260928 21:54:21.248497 7953 raft_consensus.cc:374] T 00000000000000000000000000000000 P 700a574bc5f54e5eb7a32f5d13fcc867 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:54:21.248582 7953 raft_consensus.cc:729] T 00000000000000000000000000000000 P 700a574bc5f54e5eb7a32f5d13fcc867 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 700a574bc5f54e5eb7a32f5d13fcc867, State: Initialized, Role: FOLLOWER
I20260928 21:54:21.248732 7953 consensus_queue.cc:260] T 00000000000000000000000000000000 P 700a574bc5f54e5eb7a32f5d13fcc867 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "700a574bc5f54e5eb7a32f5d13fcc867" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43129 } }
I20260928 21:54:21.248827 7953 raft_consensus.cc:388] T 00000000000000000000000000000000 P 700a574bc5f54e5eb7a32f5d13fcc867 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:54:21.248859 7953 raft_consensus.cc:482] T 00000000000000000000000000000000 P 700a574bc5f54e5eb7a32f5d13fcc867 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:54:21.248906 7953 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 700a574bc5f54e5eb7a32f5d13fcc867 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:54:21.249735 7953 raft_consensus.cc:504] T 00000000000000000000000000000000 P 700a574bc5f54e5eb7a32f5d13fcc867 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "700a574bc5f54e5eb7a32f5d13fcc867" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43129 } }
I20260928 21:54:21.249850 7953 leader_election.cc:302] T 00000000000000000000000000000000 P 700a574bc5f54e5eb7a32f5d13fcc867 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 700a574bc5f54e5eb7a32f5d13fcc867; no voters:
I20260928 21:54:21.250026 7953 leader_election.cc:288] T 00000000000000000000000000000000 P 700a574bc5f54e5eb7a32f5d13fcc867 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:54:21.250079 7958 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 700a574bc5f54e5eb7a32f5d13fcc867 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:54:21.250228 7958 raft_consensus.cc:686] T 00000000000000000000000000000000 P 700a574bc5f54e5eb7a32f5d13fcc867 [term 1 LEADER]: Becoming Leader. State: Replica: 700a574bc5f54e5eb7a32f5d13fcc867, State: Running, Role: LEADER
I20260928 21:54:21.250262 7953 sys_catalog.cc:564] T 00000000000000000000000000000000 P 700a574bc5f54e5eb7a32f5d13fcc867 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:54:21.250317 7958 consensus_queue.cc:237] T 00000000000000000000000000000000 P 700a574bc5f54e5eb7a32f5d13fcc867 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "700a574bc5f54e5eb7a32f5d13fcc867" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43129 } }
I20260928 21:54:21.250654 7958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 700a574bc5f54e5eb7a32f5d13fcc867 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 700a574bc5f54e5eb7a32f5d13fcc867. Latest consensus state: current_term: 1 leader_uuid: "700a574bc5f54e5eb7a32f5d13fcc867" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "700a574bc5f54e5eb7a32f5d13fcc867" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43129 } } }
I20260928 21:54:21.250676 7959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 700a574bc5f54e5eb7a32f5d13fcc867 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "700a574bc5f54e5eb7a32f5d13fcc867" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "700a574bc5f54e5eb7a32f5d13fcc867" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43129 } } }
I20260928 21:54:21.250725 7958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 700a574bc5f54e5eb7a32f5d13fcc867 [sys.catalog]: This master's current role is: LEADER
I20260928 21:54:21.250749 7959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 700a574bc5f54e5eb7a32f5d13fcc867 [sys.catalog]: This master's current role is: LEADER
I20260928 21:54:21.250916 7965 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:54:21.251435 7965 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:54:21.253120 7965 catalog_manager.cc:1347] Generated new cluster ID: 56bcbddfc73b4b59849a68c110070175
I20260928 21:54:21.253178 7965 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:54:21.253890 7957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:21.247721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55760 (local address 127.2.182.126:43129)
0928 21:54:21.247933 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:21.247941 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:21.247966 (+ 25us) server_negotiation.cc:407] Connection header received
0928 21:54:21.248303 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:21.248314 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:21.248384 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:21.248539 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:21.248763 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:21.250497 (+ 1734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:21.251715 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:21.252287 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:21.252415 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:21.253178 (+ 763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:21.253182 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:21.253182 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:54:21.253393 (+ 211us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:21.253434 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:54:21.253448 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:21.253505 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:21.253595 (+ 90us) server_negotiation.cc:299] Negotiation successful
0928 21:54:21.253685 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":90,"threads_started":1}
I20260928 21:54:21.254732 2777 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:54:21.272646 7965 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:54:21.273412 7965 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:54:21.276952 7965 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 700a574bc5f54e5eb7a32f5d13fcc867: Generated new TSK 0
I20260928 21:54:21.277148 7965 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 21:54:21.307801 7983 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260928 21:54:21.307911 7983 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 21:54:21.308178 7957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:21.285283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55762 (local address 127.2.182.126:43129)
0928 21:54:21.285340 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:21.285342 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:21.285435 (+ 93us) server_negotiation.cc:407] Connection header received
0928 21:54:21.285548 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:21.285549 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:21.285561 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:21.285613 (+ 52us) server_negotiation.cc:226] Negotiated authn=JWT
0928 21:54:21.285759 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:21.286414 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:21.287203 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:21.287459 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:21.287511 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:21.287723 (+ 212us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 21:54:21.307847 (+ 20124us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39513/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 21:54:21.308105 (+ 258us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55762: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39513/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":27}
W20260928 21:54:21.308234 7957 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55762: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39513/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260928 21:54:21.308637 7982 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:54:21.285179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:43129 (local address 127.0.0.1:55762)
0928 21:54:21.285355 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:21.285366 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:21.285436 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:21.285628 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:21.285629 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:21.285630 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0928 21:54:21.285696 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:21.285697 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:21.286497 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:21.286498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:21.287115 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:21.287117 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:21.287531 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:21.287532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:21.287594 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:21.287615 (+ 21us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 21:54:21.308109 (+ 20494us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39513/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 21:54:21.308236 (+ 127us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.182.126:43129: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39513/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":86,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.2.182.126:43129: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39513/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260928 21:54:21.309183 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 7886
2026-09-28T21:54:21Z chronyd exiting
I20260928 21:54:21.345408 2777 test_util.cc:175] -----------------------------------------------
I20260928 21:54:21.345494 2777 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790632440055392-2777-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (235 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260928 21:54:21.346330 2777 mini_oidc.cc:152] Starting JWKS server
I20260928 21:54:21.347005 2777 webserver.cc:460] Webserver started at https://127.0.0.1:38259/ using document root <none> and password file <none>
I20260928 21:54:21.347052 2777 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 21:54:21.347159 2777 webserver.cc:460] Webserver started at http://127.0.0.1:36547/ using document root <none> and password file <none>
2026-09-28T21:54:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:54:21Z Disabled control of system clock
I20260928 21:54:21.354950 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790632440055392-2777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790632440055392-2777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790632440055392-2777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:38337
--webserver_interface=127.2.182.126
--webserver_port=0
--builtin_ntp_servers=127.2.182.84:41649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.126:38337
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38259/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790632440055392-2777-0/testchainca.cert with env {}
W20260928 21:54:21.437309 7989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:54:21.437520 7989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:54:21.437539 7989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:54:21.438774 7989 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:54:21.438822 7989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:54:21.438833 7989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:54:21.438845 7989 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:54:21.438856 7989 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 21:54:21.438866 7989 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 21:54:21.438877 7989 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38259/jwks.json
I20260928 21:54:21.440518 7989 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:41649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790632440055392-2777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790632440055392-2777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.126:38337
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:38337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38259/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790632440055392-2777-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790632440055392-2777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:54:21.440892 7989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:54:21.441110 7989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:54:21.446135 7995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:21.446139 7997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:21.446139 7994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:21.446499 7989 server_base.cc:1034] running on GCE node
I20260928 21:54:21.446758 7989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:21.447044 7989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:21.448197 7989 hybrid_clock.cc:648] HybridClock initialized: now 1790632461448118 us; error 96 us; skew 500 ppm
I20260928 21:54:21.449697 7989 webserver.cc:460] Webserver started at http://127.2.182.126:46237/ using document root <none> and password file <none>
I20260928 21:54:21.449946 7989 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:21.449993 7989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:21.450110 7989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:54:21.451184 7989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790632440055392-2777-0/minicluster-data/master-0/data/instance:
uuid: "68d174ebfa6f4af983c6bb620091f8ac"
format_stamp: "Formatted at 2026-09-28 21:54:21 on dist-test-slave-k6lz"
server_key: "3bffc1b6a22fc99d624a2dd736c3f690"
server_key_iv: "7ac28c97bbb547b1cd50436a52deb4b0"
server_key_version: "encryptionkey@0"
I20260928 21:54:21.451539 7989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790632440055392-2777-0/minicluster-data/master-0/wal/instance:
uuid: "68d174ebfa6f4af983c6bb620091f8ac"
format_stamp: "Formatted at 2026-09-28 21:54:21 on dist-test-slave-k6lz"
server_key: "3bffc1b6a22fc99d624a2dd736c3f690"
server_key_iv: "7ac28c97bbb547b1cd50436a52deb4b0"
server_key_version: "encryptionkey@0"
I20260928 21:54:21.453014 7989 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:54:21.453817 8003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:21.454056 7989 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:54:21.454154 7989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790632440055392-2777-0/minicluster-data/master-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790632440055392-2777-0/minicluster-data/master-0/wal
uuid: "68d174ebfa6f4af983c6bb620091f8ac"
format_stamp: "Formatted at 2026-09-28 21:54:21 on dist-test-slave-k6lz"
server_key: "3bffc1b6a22fc99d624a2dd736c3f690"
server_key_iv: "7ac28c97bbb547b1cd50436a52deb4b0"
server_key_version: "encryptionkey@0"
I20260928 21:54:21.454226 7989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790632440055392-2777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790632440055392-2777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790632440055392-2777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:21.461182 7989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:54:21.461542 7989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:54:21.461689 7989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:54:21.465989 7989 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.126:38337
I20260928 21:54:21.466033 8055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.126:38337 every 8 connection(s)
I20260928 21:54:21.466349 7989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
I20260928 21:54:21.467425 8056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:54:21.469285 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 7989
I20260928 21:54:21.469398 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790632440055392-2777-0/minicluster-data/master-0/wal/instance
I20260928 21:54:21.469570 2777 external_mini_cluster.cc:1428] Setting key 11d5eb9c8805e3b7486007fd1ce9dcba
I20260928 21:54:21.469828 8056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68d174ebfa6f4af983c6bb620091f8ac: Bootstrap starting.
I20260928 21:54:21.470463 8056 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68d174ebfa6f4af983c6bb620091f8ac: Neither blocks nor log segments found. Creating new log.
I20260928 21:54:21.470731 8056 log.cc:824] T 00000000000000000000000000000000 P 68d174ebfa6f4af983c6bb620091f8ac: Log is configured to *not* fsync() on all Append() calls
I20260928 21:54:21.471493 8056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68d174ebfa6f4af983c6bb620091f8ac: No bootstrap required, opened a new log
I20260928 21:54:21.473801 8056 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68d174ebfa6f4af983c6bb620091f8ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68d174ebfa6f4af983c6bb620091f8ac" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 38337 } }
I20260928 21:54:21.473968 8056 raft_consensus.cc:374] T 00000000000000000000000000000000 P 68d174ebfa6f4af983c6bb620091f8ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:54:21.474002 8056 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68d174ebfa6f4af983c6bb620091f8ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68d174ebfa6f4af983c6bb620091f8ac, State: Initialized, Role: FOLLOWER
I20260928 21:54:21.474133 8056 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68d174ebfa6f4af983c6bb620091f8ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68d174ebfa6f4af983c6bb620091f8ac" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 38337 } }
I20260928 21:54:21.474221 8056 raft_consensus.cc:388] T 00000000000000000000000000000000 P 68d174ebfa6f4af983c6bb620091f8ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:54:21.474251 8056 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68d174ebfa6f4af983c6bb620091f8ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:54:21.474294 8056 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68d174ebfa6f4af983c6bb620091f8ac [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:54:21.474964 8056 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68d174ebfa6f4af983c6bb620091f8ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68d174ebfa6f4af983c6bb620091f8ac" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 38337 } }
I20260928 21:54:21.475072 8056 leader_election.cc:302] T 00000000000000000000000000000000 P 68d174ebfa6f4af983c6bb620091f8ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68d174ebfa6f4af983c6bb620091f8ac; no voters:
I20260928 21:54:21.475260 8056 leader_election.cc:288] T 00000000000000000000000000000000 P 68d174ebfa6f4af983c6bb620091f8ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:54:21.475303 8061 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68d174ebfa6f4af983c6bb620091f8ac [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:54:21.475430 8061 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68d174ebfa6f4af983c6bb620091f8ac [term 1 LEADER]: Becoming Leader. State: Replica: 68d174ebfa6f4af983c6bb620091f8ac, State: Running, Role: LEADER
I20260928 21:54:21.475510 8061 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68d174ebfa6f4af983c6bb620091f8ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68d174ebfa6f4af983c6bb620091f8ac" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 38337 } }
I20260928 21:54:21.475544 8056 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68d174ebfa6f4af983c6bb620091f8ac [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:54:21.475615 8059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:21.470037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52582 (local address 127.2.182.126:38337)
0928 21:54:21.470244 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:21.470252 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:21.470281 (+ 29us) server_negotiation.cc:407] Connection header received
0928 21:54:21.470784 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:21.470794 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:21.470868 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:21.471009 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:21.471193 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:21.472213 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:21.473397 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:21.473896 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:21.474025 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:21.474701 (+ 676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:21.474705 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:21.474705 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:54:21.474880 (+ 175us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:21.474948 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:54:21.474961 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:21.475024 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:21.475249 (+ 225us) server_negotiation.cc:299] Negotiation successful
0928 21:54:21.475381 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":82,"threads_started":1}
I20260928 21:54:21.475811 8061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68d174ebfa6f4af983c6bb620091f8ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68d174ebfa6f4af983c6bb620091f8ac. Latest consensus state: current_term: 1 leader_uuid: "68d174ebfa6f4af983c6bb620091f8ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68d174ebfa6f4af983c6bb620091f8ac" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 38337 } } }
I20260928 21:54:21.475875 8061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68d174ebfa6f4af983c6bb620091f8ac [sys.catalog]: This master's current role is: LEADER
I20260928 21:54:21.475970 8062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68d174ebfa6f4af983c6bb620091f8ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "68d174ebfa6f4af983c6bb620091f8ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68d174ebfa6f4af983c6bb620091f8ac" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 38337 } } }
I20260928 21:54:21.476056 8062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68d174ebfa6f4af983c6bb620091f8ac [sys.catalog]: This master's current role is: LEADER
I20260928 21:54:21.476647 2777 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260928 21:54:21.477458 8077 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 68d174ebfa6f4af983c6bb620091f8ac: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:54:21.477515 8077 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:54:21.477555 8065 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:54:21.477705 8065 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:54:21.479185 8065 catalog_manager.cc:1347] Generated new cluster ID: dd590f092b0249e8b90d1102d71e9cf8
I20260928 21:54:21.479242 8065 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:54:21.489936 8065 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:54:21.490477 8065 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:54:21.499096 8065 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 68d174ebfa6f4af983c6bb620091f8ac: Generated new TSK 0
I20260928 21:54:21.499387 8065 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 21:54:21.525221 8086 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260928 21:54:21.525322 8086 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 21:54:21.525624 8059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:21.508546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52588 (local address 127.2.182.126:38337)
0928 21:54:21.508594 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:21.508596 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:21.508673 (+ 77us) server_negotiation.cc:407] Connection header received
0928 21:54:21.508746 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:21.508747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:21.508757 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:21.508831 (+ 74us) server_negotiation.cc:226] Negotiated authn=JWT
0928 21:54:21.509001 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:21.509523 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:21.510710 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:21.511013 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:21.511116 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:21.511423 (+ 307us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 21:54:21.525422 (+ 13999us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38259/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 21:54:21.525561 (+ 139us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52588: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38259/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":26}
W20260928 21:54:21.525668 8059 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52588: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38259/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260928 21:54:21.525799 8085 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:54:21.508455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:38337 (local address 127.0.0.1:52588)
0928 21:54:21.508596 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:21.508608 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:21.508682 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:21.508836 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:21.508836 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:21.508838 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 21:54:21.508899 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:21.508901 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:21.509594 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:21.509595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:21.510542 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:21.510545 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:21.511130 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:21.511131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:21.511200 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:21.511207 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 21:54:21.525552 (+ 14345us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38259/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 21:54:21.525700 (+ 148us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.182.126:38337: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38259/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":81,"threads_started":1}
I20260928 21:54:21.526078 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 7989
2026-09-28T21:54:21Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (218 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260928 21:54:21.564719 2777 mini_oidc.cc:152] Starting JWKS server
I20260928 21:54:21.565492 2777 webserver.cc:460] Webserver started at https://127.0.0.1:36733/ using document root <none> and password file <none>
I20260928 21:54:21.565536 2777 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 21:54:21.565649 2777 webserver.cc:460] Webserver started at http://127.0.0.1:37509/ using document root <none> and password file <none>
2026-09-28T21:54:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:54:21Z Disabled control of system clock
I20260928 21:54:21.574909 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790632440055392-2777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790632440055392-2777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790632440055392-2777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:43693
--webserver_interface=127.2.182.126
--webserver_port=0
--builtin_ntp_servers=127.2.182.84:42515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.126:43693
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36733/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 21:54:21.661002 8092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:54:21.661252 8092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:54:21.661284 8092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:54:21.662554 8092 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:54:21.662606 8092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:54:21.662621 8092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:54:21.662638 8092 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:54:21.662653 8092 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 21:54:21.662668 8092 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 21:54:21.662683 8092 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36733/jwks.json
I20260928 21:54:21.664187 8092 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:42515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790632440055392-2777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790632440055392-2777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.126:43693
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:43693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36733/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790632440055392-2777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:54:21.664476 8092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:54:21.664778 8092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:54:21.669860 8098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:21.669941 8097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:21.669862 8100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:21.669895 8092 server_base.cc:1034] running on GCE node
I20260928 21:54:21.670310 8092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:21.670591 8092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:21.671763 8092 hybrid_clock.cc:648] HybridClock initialized: now 1790632461671738 us; error 51 us; skew 500 ppm
I20260928 21:54:21.673177 8092 webserver.cc:460] Webserver started at http://127.2.182.126:42169/ using document root <none> and password file <none>
I20260928 21:54:21.673467 8092 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:21.673532 8092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:21.673679 8092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:54:21.674773 8092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790632440055392-2777-0/minicluster-data/master-0/data/instance:
uuid: "4549abd4791d4505b37adb75643e5f1d"
format_stamp: "Formatted at 2026-09-28 21:54:21 on dist-test-slave-k6lz"
server_key: "1cd66c3a739075019c91a32338a95627"
server_key_iv: "eaa64a7b943211c77ee648f7d4c535b7"
server_key_version: "encryptionkey@0"
I20260928 21:54:21.675114 8092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790632440055392-2777-0/minicluster-data/master-0/wal/instance:
uuid: "4549abd4791d4505b37adb75643e5f1d"
format_stamp: "Formatted at 2026-09-28 21:54:21 on dist-test-slave-k6lz"
server_key: "1cd66c3a739075019c91a32338a95627"
server_key_iv: "eaa64a7b943211c77ee648f7d4c535b7"
server_key_version: "encryptionkey@0"
I20260928 21:54:21.676470 8092 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:54:21.677373 8106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:21.677608 8092 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:54:21.677700 8092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790632440055392-2777-0/minicluster-data/master-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790632440055392-2777-0/minicluster-data/master-0/wal
uuid: "4549abd4791d4505b37adb75643e5f1d"
format_stamp: "Formatted at 2026-09-28 21:54:21 on dist-test-slave-k6lz"
server_key: "1cd66c3a739075019c91a32338a95627"
server_key_iv: "eaa64a7b943211c77ee648f7d4c535b7"
server_key_version: "encryptionkey@0"
I20260928 21:54:21.677774 8092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790632440055392-2777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790632440055392-2777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790632440055392-2777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:21.697777 8092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:54:21.698129 8092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:54:21.698274 8092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:54:21.703083 8092 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.126:43693
I20260928 21:54:21.703117 8158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.126:43693 every 8 connection(s)
I20260928 21:54:21.703492 8092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
I20260928 21:54:21.704164 8159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:54:21.707028 8159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4549abd4791d4505b37adb75643e5f1d: Bootstrap starting.
I20260928 21:54:21.707679 8159 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4549abd4791d4505b37adb75643e5f1d: Neither blocks nor log segments found. Creating new log.
I20260928 21:54:21.707959 8159 log.cc:824] T 00000000000000000000000000000000 P 4549abd4791d4505b37adb75643e5f1d: Log is configured to *not* fsync() on all Append() calls
I20260928 21:54:21.708891 8159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4549abd4791d4505b37adb75643e5f1d: No bootstrap required, opened a new log
I20260928 21:54:21.710664 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 8092
I20260928 21:54:21.710739 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790632440055392-2777-0/minicluster-data/master-0/wal/instance
I20260928 21:54:21.710608 8159 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4549abd4791d4505b37adb75643e5f1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4549abd4791d4505b37adb75643e5f1d" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43693 } }
I20260928 21:54:21.710774 8159 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4549abd4791d4505b37adb75643e5f1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:54:21.710819 8159 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4549abd4791d4505b37adb75643e5f1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4549abd4791d4505b37adb75643e5f1d, State: Initialized, Role: FOLLOWER
I20260928 21:54:21.710898 2777 external_mini_cluster.cc:1428] Setting key 36fc461059ba5f2bb6bb890912837c0d
I20260928 21:54:21.710974 8159 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4549abd4791d4505b37adb75643e5f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4549abd4791d4505b37adb75643e5f1d" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43693 } }
I20260928 21:54:21.711062 8159 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4549abd4791d4505b37adb75643e5f1d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:54:21.711095 8159 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4549abd4791d4505b37adb75643e5f1d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:54:21.711149 8159 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4549abd4791d4505b37adb75643e5f1d [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:54:21.711982 8159 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4549abd4791d4505b37adb75643e5f1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4549abd4791d4505b37adb75643e5f1d" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43693 } }
I20260928 21:54:21.712096 8159 leader_election.cc:302] T 00000000000000000000000000000000 P 4549abd4791d4505b37adb75643e5f1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4549abd4791d4505b37adb75643e5f1d; no voters:
I20260928 21:54:21.712261 8159 leader_election.cc:288] T 00000000000000000000000000000000 P 4549abd4791d4505b37adb75643e5f1d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:54:21.712292 8164 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4549abd4791d4505b37adb75643e5f1d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:54:21.712391 8164 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4549abd4791d4505b37adb75643e5f1d [term 1 LEADER]: Becoming Leader. State: Replica: 4549abd4791d4505b37adb75643e5f1d, State: Running, Role: LEADER
I20260928 21:54:21.712462 8164 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4549abd4791d4505b37adb75643e5f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4549abd4791d4505b37adb75643e5f1d" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43693 } }
I20260928 21:54:21.712534 8159 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4549abd4791d4505b37adb75643e5f1d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:54:21.712821 8165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4549abd4791d4505b37adb75643e5f1d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4549abd4791d4505b37adb75643e5f1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4549abd4791d4505b37adb75643e5f1d" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43693 } } }
I20260928 21:54:21.712925 8165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4549abd4791d4505b37adb75643e5f1d [sys.catalog]: This master's current role is: LEADER
I20260928 21:54:21.713086 8166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4549abd4791d4505b37adb75643e5f1d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4549abd4791d4505b37adb75643e5f1d. Latest consensus state: current_term: 1 leader_uuid: "4549abd4791d4505b37adb75643e5f1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4549abd4791d4505b37adb75643e5f1d" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43693 } } }
I20260928 21:54:21.713145 8166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4549abd4791d4505b37adb75643e5f1d [sys.catalog]: This master's current role is: LEADER
I20260928 21:54:21.713248 8170 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:54:21.713784 8170 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:54:21.715451 8170 catalog_manager.cc:1347] Generated new cluster ID: 6fcef583e7a440939e39250313e840a0
I20260928 21:54:21.715507 8170 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:54:21.717772 8163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:21.711434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36684 (local address 127.2.182.126:43693)
0928 21:54:21.711683 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:21.711694 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:21.711732 (+ 38us) server_negotiation.cc:407] Connection header received
0928 21:54:21.711855 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:21.711862 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:21.711955 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:21.712108 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:21.712362 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:21.714284 (+ 1922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:21.715437 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:21.716018 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:21.716141 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:21.716911 (+ 770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:21.716915 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:21.716915 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:54:21.717122 (+ 207us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:21.717165 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:54:21.717181 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:21.717254 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:21.717414 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 21:54:21.717507 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":112,"threads_started":1}
I20260928 21:54:21.718770 2777 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:54:21.730767 8170 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:54:21.731531 8170 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:54:21.752235 8170 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4549abd4791d4505b37adb75643e5f1d: Generated new TSK 0
I20260928 21:54:21.752460 8170 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:54:21.763824 8163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:21.758871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36692 (local address 127.2.182.126:43693)
0928 21:54:21.758913 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:21.758915 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:21.759174 (+ 259us) server_negotiation.cc:407] Connection header received
0928 21:54:21.759348 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:21.759349 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:21.759365 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:21.759436 (+ 71us) server_negotiation.cc:226] Negotiated authn=JWT
0928 21:54:21.759629 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:21.760472 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:21.761864 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:21.762208 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:21.762280 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:21.762618 (+ 338us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 21:54:21.763603 (+ 985us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36733/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 21:54:21.763747 (+ 144us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36692: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36733/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 21:54:21.763891 8163 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36692: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36733/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260928 21:54:21.763907 8188 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:54:21.758814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:43693 (local address 127.0.0.1:36692)
0928 21:54:21.759103 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:21.759113 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:21.759209 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:21.759481 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:21.759482 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:21.759484 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 21:54:21.759562 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:21.759564 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:21.760569 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:21.760570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:21.761747 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:21.761751 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:21.762345 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:21.762346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:21.762431 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:21.762438 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 21:54:21.763719 (+ 1281us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36733/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 21:54:21.763819 (+ 100us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.182.126:43693: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36733/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":174,"threads_started":1}
I20260928 21:54:21.764192 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 8092
E20260928 21:54:21.776495 8189 webserver.cc:533] Webserver: sslize failed:
E20260928 21:54:21.776589 8189 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-28T21:54:21Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (237 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-28T21:54:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:54:21Z Disabled control of system clock
I20260928 21:54:21.809988 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790632440055392-2777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790632440055392-2777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790632440055392-2777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:39253
--webserver_interface=127.2.182.126
--webserver_port=0
--builtin_ntp_servers=127.2.182.84:42791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.126:39253
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790632440055392-2777-0/testchainca.cert with env {}
W20260928 21:54:21.891474 8193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:54:21.891701 8193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:54:21.891731 8193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:54:21.893026 8193 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:54:21.893077 8193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:54:21.893100 8193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:54:21.893115 8193 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:54:21.893131 8193 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 21:54:21.893150 8193 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 21:54:21.893172 8193 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260928 21:54:21.895067 8193 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:42791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790632440055392-2777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790632440055392-2777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.126:39253
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:39253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790632440055392-2777-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790632440055392-2777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:54:21.895406 8193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:54:21.895633 8193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:54:21.900769 8199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:21.900899 8198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:21.900981 8201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:21.901252 8193 server_base.cc:1034] running on GCE node
I20260928 21:54:21.901544 8193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:21.902009 8193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:21.903214 8193 hybrid_clock.cc:648] HybridClock initialized: now 1790632461903196 us; error 34 us; skew 500 ppm
I20260928 21:54:21.904594 8193 webserver.cc:460] Webserver started at http://127.2.182.126:43369/ using document root <none> and password file <none>
I20260928 21:54:21.904906 8193 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:21.904966 8193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:21.905089 8193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:54:21.906183 8193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790632440055392-2777-0/minicluster-data/master-0/data/instance:
uuid: "d422ba73014f4c819d960197fe0d764e"
format_stamp: "Formatted at 2026-09-28 21:54:21 on dist-test-slave-k6lz"
server_key: "7845ba2580c9afe780d112afe012aa38"
server_key_iv: "3c542e728a9331de77487596b732f4eb"
server_key_version: "encryptionkey@0"
I20260928 21:54:21.906523 8193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790632440055392-2777-0/minicluster-data/master-0/wal/instance:
uuid: "d422ba73014f4c819d960197fe0d764e"
format_stamp: "Formatted at 2026-09-28 21:54:21 on dist-test-slave-k6lz"
server_key: "7845ba2580c9afe780d112afe012aa38"
server_key_iv: "3c542e728a9331de77487596b732f4eb"
server_key_version: "encryptionkey@0"
I20260928 21:54:21.907917 8193 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 21:54:21.908639 8207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:21.908776 8193 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:21.908866 8193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790632440055392-2777-0/minicluster-data/master-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790632440055392-2777-0/minicluster-data/master-0/wal
uuid: "d422ba73014f4c819d960197fe0d764e"
format_stamp: "Formatted at 2026-09-28 21:54:21 on dist-test-slave-k6lz"
server_key: "7845ba2580c9afe780d112afe012aa38"
server_key_iv: "3c542e728a9331de77487596b732f4eb"
server_key_version: "encryptionkey@0"
I20260928 21:54:21.908927 8193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790632440055392-2777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790632440055392-2777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790632440055392-2777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:21.918678 8193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:54:21.918985 8193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:54:21.919107 8193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:54:21.923019 8193 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.126:39253
I20260928 21:54:21.923081 8259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.126:39253 every 8 connection(s)
I20260928 21:54:21.923604 8193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
I20260928 21:54:21.924872 8260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:54:21.924901 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 8193
I20260928 21:54:21.925006 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790632440055392-2777-0/minicluster-data/master-0/wal/instance
I20260928 21:54:21.925184 2777 external_mini_cluster.cc:1428] Setting key 526f900faae385cdaafb3885ca388012
I20260928 21:54:21.927892 8260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d422ba73014f4c819d960197fe0d764e: Bootstrap starting.
I20260928 21:54:21.928617 8260 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d422ba73014f4c819d960197fe0d764e: Neither blocks nor log segments found. Creating new log.
I20260928 21:54:21.928875 8260 log.cc:824] T 00000000000000000000000000000000 P d422ba73014f4c819d960197fe0d764e: Log is configured to *not* fsync() on all Append() calls
I20260928 21:54:21.929520 8260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d422ba73014f4c819d960197fe0d764e: No bootstrap required, opened a new log
I20260928 21:54:21.931604 8263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:21.925977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60518 (local address 127.2.182.126:39253)
0928 21:54:21.926200 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:21.926208 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:21.926240 (+ 32us) server_negotiation.cc:407] Connection header received
0928 21:54:21.926691 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:21.926698 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:21.926843 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:21.926967 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:21.927396 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:21.928909 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:21.929761 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:21.930203 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:21.930323 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:21.930975 (+ 652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:21.930979 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:21.930980 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:54:21.931140 (+ 160us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:21.931172 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:54:21.931183 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:21.931247 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:21.931310 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 21:54:21.931432 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":69,"threads_started":1}
I20260928 21:54:21.931617 8260 raft_consensus.cc:348] T 00000000000000000000000000000000 P d422ba73014f4c819d960197fe0d764e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d422ba73014f4c819d960197fe0d764e" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 39253 } }
I20260928 21:54:21.931833 8260 raft_consensus.cc:374] T 00000000000000000000000000000000 P d422ba73014f4c819d960197fe0d764e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:54:21.931880 8260 raft_consensus.cc:729] T 00000000000000000000000000000000 P d422ba73014f4c819d960197fe0d764e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d422ba73014f4c819d960197fe0d764e, State: Initialized, Role: FOLLOWER
I20260928 21:54:21.931991 8260 consensus_queue.cc:260] T 00000000000000000000000000000000 P d422ba73014f4c819d960197fe0d764e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d422ba73014f4c819d960197fe0d764e" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 39253 } }
I20260928 21:54:21.932045 8260 raft_consensus.cc:388] T 00000000000000000000000000000000 P d422ba73014f4c819d960197fe0d764e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:54:21.932071 8260 raft_consensus.cc:482] T 00000000000000000000000000000000 P d422ba73014f4c819d960197fe0d764e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:54:21.932116 8260 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d422ba73014f4c819d960197fe0d764e [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:54:21.932458 2777 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:54:21.932823 8260 raft_consensus.cc:504] T 00000000000000000000000000000000 P d422ba73014f4c819d960197fe0d764e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d422ba73014f4c819d960197fe0d764e" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 39253 } }
I20260928 21:54:21.932941 8260 leader_election.cc:302] T 00000000000000000000000000000000 P d422ba73014f4c819d960197fe0d764e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d422ba73014f4c819d960197fe0d764e; no voters:
I20260928 21:54:21.933115 8260 leader_election.cc:288] T 00000000000000000000000000000000 P d422ba73014f4c819d960197fe0d764e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:54:21.933166 8265 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d422ba73014f4c819d960197fe0d764e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:54:21.933315 8265 raft_consensus.cc:686] T 00000000000000000000000000000000 P d422ba73014f4c819d960197fe0d764e [term 1 LEADER]: Becoming Leader. State: Replica: d422ba73014f4c819d960197fe0d764e, State: Running, Role: LEADER
I20260928 21:54:21.933391 8260 sys_catalog.cc:564] T 00000000000000000000000000000000 P d422ba73014f4c819d960197fe0d764e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:54:21.933399 8265 consensus_queue.cc:237] T 00000000000000000000000000000000 P d422ba73014f4c819d960197fe0d764e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d422ba73014f4c819d960197fe0d764e" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 39253 } }
I20260928 21:54:21.933794 8267 sys_catalog.cc:455] T 00000000000000000000000000000000 P d422ba73014f4c819d960197fe0d764e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d422ba73014f4c819d960197fe0d764e. Latest consensus state: current_term: 1 leader_uuid: "d422ba73014f4c819d960197fe0d764e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d422ba73014f4c819d960197fe0d764e" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 39253 } } }
I20260928 21:54:21.933808 8266 sys_catalog.cc:455] T 00000000000000000000000000000000 P d422ba73014f4c819d960197fe0d764e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d422ba73014f4c819d960197fe0d764e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d422ba73014f4c819d960197fe0d764e" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 39253 } } }
I20260928 21:54:21.933907 8267 sys_catalog.cc:458] T 00000000000000000000000000000000 P d422ba73014f4c819d960197fe0d764e [sys.catalog]: This master's current role is: LEADER
I20260928 21:54:21.933914 8266 sys_catalog.cc:458] T 00000000000000000000000000000000 P d422ba73014f4c819d960197fe0d764e [sys.catalog]: This master's current role is: LEADER
I20260928 21:54:21.934137 8272 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:54:21.935165 8272 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:54:21.937067 8272 catalog_manager.cc:1347] Generated new cluster ID: 661ec5c809eb4d35941cfc02fca210dc
I20260928 21:54:21.937120 8272 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:54:21.943814 8272 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:54:21.944551 8272 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:54:21.952657 8272 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d422ba73014f4c819d960197fe0d764e: Generated new TSK 0
I20260928 21:54:21.952899 8272 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:54:22.394431 8263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:21.958783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60534 (local address 127.2.182.126:39253)
0928 21:54:21.958843 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:21.958845 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:21.958856 (+ 11us) server_negotiation.cc:407] Connection header received
0928 21:54:21.958942 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:21.958943 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:21.958957 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:21.959031 (+ 74us) server_negotiation.cc:226] Negotiated authn=JWT
0928 21:54:21.959216 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:21.960036 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:21.961135 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:21.961474 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:21.961568 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:21.961853 (+ 285us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 21:54:22.394135 (+432282us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 21:54:22.394356 (+ 221us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60534: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":35}
W20260928 21:54:22.394567 8289 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:54:21.958660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:39253 (local address 127.0.0.1:60534)
0928 21:54:21.958793 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:21.958804 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:21.958856 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:21.959076 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:21.959077 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:21.959079 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 21:54:21.959143 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:21.959145 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:21.960233 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:21.960235 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:21.961005 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:21.961008 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:21.961649 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:21.961650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:21.961718 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:21.961727 (+ 9us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 21:54:22.394331 (+432604us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 21:54:22.394469 (+ 138us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.182.126:39253: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":67,"threads_started":1}
W20260928 21:54:22.394505 8263 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60534: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
I20260928 21:54:22.394850 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 8193
2026-09-28T21:54:22Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (610 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (92 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (96 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:54:24 dist-test-slave-k6lz krb5kdc[8301](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:54:24 dist-test-slave-k6lz krb5kdc[8301](info): set up 2 sockets
Sep 28 21:54:24 dist-test-slave-k6lz krb5kdc[8301](info): commencing operation
krb5kdc: starting...
W20260928 21:54:25.585282 2777 mini_kdc.cc:121] Time spent starting KDC: real 2.982s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260928 21:54:26.001283 2777 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.416s user 0.000s sys 0.002s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260928 21:54:26.299468 2777 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.298s user 0.000s sys 0.002s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260928 21:54:26.423237 2777 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.124s user 0.000s sys 0.001s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/krb5kdc/test-user.keytab.
W20260928 21:54:26.808082 2777 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.385s user 0.000s sys 0.001s
Sep 28 21:54:26 dist-test-slave-k6lz krb5kdc[8301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632466, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:54:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:54:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab.
Entry for principal kudu/127.2.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab.
I20260928 21:54:26.850770 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:41921
--webserver_interface=127.2.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab
--principal=kudu/127.2.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:37835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.126:41921
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:54:26.932132 8314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:54:26.932351 8314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:54:26.932372 8314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:54:26.933523 8314 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:54:26.933559 8314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:54:26.933573 8314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:54:26.933583 8314 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:54:26.933593 8314 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:54:26.934993 8314 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:37835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.126:41921
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab
--principal=kudu/127.2.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:41921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8314
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:54:26.935223 8314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:54:26.935472 8314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:54:26.940279 8319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:26.940439 8320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:26.940291 8322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:26.940558 8314 server_base.cc:1034] running on GCE node
I20260928 21:54:26.940784 8314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:26.941073 8314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:26.942245 8314 hybrid_clock.cc:648] HybridClock initialized: now 1790632466942225 us; error 32 us; skew 500 ppm
Sep 28 21:54:26 dist-test-slave-k6lz krb5kdc[8301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632466, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:54:26.944763 8314 init.cc:377] Logged in from keytab as kudu/127.2.182.126@KRBTEST.COM (short username kudu)
I20260928 21:54:26.945202 8314 webserver.cc:460] Webserver started at http://127.2.182.126:44641/ using document root <none> and password file <none>
I20260928 21:54:26.945412 8314 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:26.945454 8314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:26.945547 8314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:54:26.946606 8314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/minicluster-data/master-0/data/instance:
uuid: "8012ad3315964f48b7379bf1b427f3d1"
format_stamp: "Formatted at 2026-09-28 21:54:26 on dist-test-slave-k6lz"
server_key: "bb43d7b1be969e7e531dceb3472f8d87"
server_key_iv: "798466e2b728b1aa2558666eb3285e3e"
server_key_version: "encryptionkey@0"
I20260928 21:54:26.946897 8314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/minicluster-data/master-0/wal/instance:
uuid: "8012ad3315964f48b7379bf1b427f3d1"
format_stamp: "Formatted at 2026-09-28 21:54:26 on dist-test-slave-k6lz"
server_key: "bb43d7b1be969e7e531dceb3472f8d87"
server_key_iv: "798466e2b728b1aa2558666eb3285e3e"
server_key_version: "encryptionkey@0"
I20260928 21:54:26.948186 8314 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:54:26.948975 8329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:26.949126 8314 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:54:26.949204 8314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/minicluster-data/master-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/minicluster-data/master-0/wal
uuid: "8012ad3315964f48b7379bf1b427f3d1"
format_stamp: "Formatted at 2026-09-28 21:54:26 on dist-test-slave-k6lz"
server_key: "bb43d7b1be969e7e531dceb3472f8d87"
server_key_iv: "798466e2b728b1aa2558666eb3285e3e"
server_key_version: "encryptionkey@0"
I20260928 21:54:26.949312 8314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:26.957095 8314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:54:26.959244 8314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:54:26.959494 8314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:54:26.963227 8314 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.126:41921
I20260928 21:54:26.963271 8381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.126:41921 every 8 connection(s)
I20260928 21:54:26.963606 8314 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
I20260928 21:54:26.964438 8382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:54:26.965646 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 8314
I20260928 21:54:26.965721 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/minicluster-data/master-0/wal/instance
I20260928 21:54:26.965891 2777 external_mini_cluster.cc:1428] Setting key 9169fd9b94bcb4547937e4996d05a7ad
I20260928 21:54:26.966981 8382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8012ad3315964f48b7379bf1b427f3d1: Bootstrap starting.
I20260928 21:54:26.967702 8382 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8012ad3315964f48b7379bf1b427f3d1: Neither blocks nor log segments found. Creating new log.
I20260928 21:54:26.968034 8382 log.cc:824] T 00000000000000000000000000000000 P 8012ad3315964f48b7379bf1b427f3d1: Log is configured to *not* fsync() on all Append() calls
I20260928 21:54:26.968657 8382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8012ad3315964f48b7379bf1b427f3d1: No bootstrap required, opened a new log
I20260928 21:54:26.970889 8382 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8012ad3315964f48b7379bf1b427f3d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8012ad3315964f48b7379bf1b427f3d1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 41921 } }
I20260928 21:54:26.971040 8382 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8012ad3315964f48b7379bf1b427f3d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:54:26.971063 8382 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8012ad3315964f48b7379bf1b427f3d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8012ad3315964f48b7379bf1b427f3d1, State: Initialized, Role: FOLLOWER
I20260928 21:54:26.971217 8382 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8012ad3315964f48b7379bf1b427f3d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8012ad3315964f48b7379bf1b427f3d1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 41921 } }
I20260928 21:54:26.971315 8382 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8012ad3315964f48b7379bf1b427f3d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:54:26.971375 8382 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8012ad3315964f48b7379bf1b427f3d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:54:26.971418 8382 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8012ad3315964f48b7379bf1b427f3d1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:54:26.972187 8382 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8012ad3315964f48b7379bf1b427f3d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8012ad3315964f48b7379bf1b427f3d1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 41921 } }
I20260928 21:54:26.972309 8382 leader_election.cc:302] T 00000000000000000000000000000000 P 8012ad3315964f48b7379bf1b427f3d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8012ad3315964f48b7379bf1b427f3d1; no voters:
I20260928 21:54:26.972471 8382 leader_election.cc:288] T 00000000000000000000000000000000 P 8012ad3315964f48b7379bf1b427f3d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:54:26.972494 8387 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8012ad3315964f48b7379bf1b427f3d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:54:26.972595 8387 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8012ad3315964f48b7379bf1b427f3d1 [term 1 LEADER]: Becoming Leader. State: Replica: 8012ad3315964f48b7379bf1b427f3d1, State: Running, Role: LEADER
I20260928 21:54:26.972734 8382 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8012ad3315964f48b7379bf1b427f3d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:54:26.972741 8387 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8012ad3315964f48b7379bf1b427f3d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8012ad3315964f48b7379bf1b427f3d1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 41921 } }
Sep 28 21:54:26 dist-test-slave-k6lz krb5kdc[8301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632466, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:54:26.973133 8388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8012ad3315964f48b7379bf1b427f3d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8012ad3315964f48b7379bf1b427f3d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8012ad3315964f48b7379bf1b427f3d1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 41921 } } }
I20260928 21:54:26.973201 8388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8012ad3315964f48b7379bf1b427f3d1 [sys.catalog]: This master's current role is: LEADER
I20260928 21:54:26.973325 8389 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8012ad3315964f48b7379bf1b427f3d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8012ad3315964f48b7379bf1b427f3d1. Latest consensus state: current_term: 1 leader_uuid: "8012ad3315964f48b7379bf1b427f3d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8012ad3315964f48b7379bf1b427f3d1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 41921 } } }
I20260928 21:54:26.973374 8389 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8012ad3315964f48b7379bf1b427f3d1 [sys.catalog]: This master's current role is: LEADER
I20260928 21:54:26.973487 8393 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:54:26.974056 8393 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:54:26.975615 8393 catalog_manager.cc:1347] Generated new cluster ID: 9eaa917098534ad588faf56c1615870c
I20260928 21:54:26.975667 8393 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:54:26.977094 8385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:26.966383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45820 (local address 127.2.182.126:41921)
0928 21:54:26.966541 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:26.966549 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:26.966574 (+ 25us) server_negotiation.cc:407] Connection header received
0928 21:54:26.967003 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:26.967010 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:26.967060 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:26.967190 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:26.967964 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:26.968997 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:26.970065 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:26.970511 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:26.970598 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:26.973477 (+ 2879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:26.973482 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:26.973483 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:26.973496 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:26.975575 (+ 2079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:26.975950 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:26.975952 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:26.975952 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:26.976048 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:26.976243 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:26.976244 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:26.976244 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:26.976492 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:26.976599 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:26.976761 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 21:54:26.976907 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":65,"threads_started":1}
I20260928 21:54:26.991410 8393 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:54:26.992022 8393 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:54:26.994750 8393 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8012ad3315964f48b7379bf1b427f3d1: Generated new TSK 0
I20260928 21:54:26.994921 8393 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.182.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab.
Entry for principal kudu/127.2.182.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab.
I20260928 21:54:27.027158 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.65:0
--local_ip_for_outbound_sockets=127.2.182.65
--webserver_interface=127.2.182.65
--webserver_port=0
--tserver_master_addrs=127.2.182.126:41921
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab
--principal=kudu/127.2.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:37835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:54:27.131606 8408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:54:27.131830 8408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:54:27.131855 8408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:54:27.133092 8408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:54:27.133140 8408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:54:27.133167 8408 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.65
I20260928 21:54:27.134717 8408 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:37835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab
--principal=kudu/127.2.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.182.65
--webserver_port=0
--tserver_master_addrs=127.2.182.126:41921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.65
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:54:27.134960 8408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:54:27.135185 8408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:54:27.140237 8414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:27.140237 8416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:27.140407 8408 server_base.cc:1034] running on GCE node
W20260928 21:54:27.140342 8413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:27.140648 8408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:27.141017 8408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:27.142189 8408 hybrid_clock.cc:648] HybridClock initialized: now 1790632467142174 us; error 28 us; skew 500 ppm
Sep 28 21:54:27 dist-test-slave-k6lz krb5kdc[8301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632467, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:54:27.144834 8408 init.cc:377] Logged in from keytab as kudu/127.2.182.65@KRBTEST.COM (short username kudu)
I20260928 21:54:27.145300 8408 webserver.cc:460] Webserver started at http://127.2.182.65:35061/ using document root <none> and password file <none>
I20260928 21:54:27.145498 8408 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:27.145537 8408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:27.145622 8408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:54:27.146649 8408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/minicluster-data/ts-0/data/instance:
uuid: "cec5cdc1cb7b43fc83f3c6530cd0841b"
format_stamp: "Formatted at 2026-09-28 21:54:27 on dist-test-slave-k6lz"
server_key: "2c315e884b574c504117305b4af30913"
server_key_iv: "0f4d5f976d1a779449f7fabbf15fcc4f"
server_key_version: "encryptionkey@0"
I20260928 21:54:27.146950 8408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/minicluster-data/ts-0/wal/instance:
uuid: "cec5cdc1cb7b43fc83f3c6530cd0841b"
format_stamp: "Formatted at 2026-09-28 21:54:27 on dist-test-slave-k6lz"
server_key: "2c315e884b574c504117305b4af30913"
server_key_iv: "0f4d5f976d1a779449f7fabbf15fcc4f"
server_key_version: "encryptionkey@0"
I20260928 21:54:27.148232 8408 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 21:54:27.149001 8423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:27.149173 8408 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 21:54:27.149257 8408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/minicluster-data/ts-0/wal
uuid: "cec5cdc1cb7b43fc83f3c6530cd0841b"
format_stamp: "Formatted at 2026-09-28 21:54:27 on dist-test-slave-k6lz"
server_key: "2c315e884b574c504117305b4af30913"
server_key_iv: "0f4d5f976d1a779449f7fabbf15fcc4f"
server_key_version: "encryptionkey@0"
I20260928 21:54:27.149317 8408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:27.184688 8408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:54:27.186812 8408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:54:27.186924 8408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:54:27.187108 8408 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:54:27.187403 8408 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:54:27.187443 8408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:27.187466 8408 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:54:27.187482 8408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:27.192975 8408 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.65:34953
I20260928 21:54:27.193042 8536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.65:34953 every 8 connection(s)
I20260928 21:54:27.193346 8408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
Sep 28 21:54:27 dist-test-slave-k6lz krb5kdc[8301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632467, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.65@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:54:27.202742 8385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:27.194179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.65:56253 (local address 127.2.182.126:41921)
0928 21:54:27.194236 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:27.194238 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:27.194520 (+ 282us) server_negotiation.cc:407] Connection header received
0928 21:54:27.195511 (+ 991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:27.195513 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:27.195525 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:27.195576 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:27.196193 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:27.196699 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:27.197692 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:27.197911 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:27.197966 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:27.200072 (+ 2106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:27.200073 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:27.200074 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:27.200082 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:27.201425 (+ 1343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:27.201805 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:27.201807 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:27.201807 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:27.201870 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:27.202084 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:27.202085 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:27.202085 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:27.202206 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:27.202261 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:27.202541 (+ 280us) server_negotiation.cc:299] Negotiation successful
0928 21:54:27.202647 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 21:54:27.202934 8539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:27.194137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:41921 (local address 127.2.182.65:56253)
0928 21:54:27.194395 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:27.194432 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:27.195398 (+ 966us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:27.195644 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:27.195647 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:27.195910 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:27.196098 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:27.196102 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:27.196789 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:27.196789 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:27.197604 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:27.197607 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:27.197985 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:27.197986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:27.198100 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:27.198597 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:27.198605 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:27.199976 (+ 1371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:27.201503 (+ 1527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:27.201511 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:27.201514 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:27.201737 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:27.201925 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:27.201926 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:27.201926 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:27.202030 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:27.202261 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:27.202268 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:27.202418 (+ 150us) client_negotiation.cc:769] Sending connection context
0928 21:54:27.202509 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 21:54:27.202681 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":76,"threads_started":1}
I20260928 21:54:27.203178 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 8408
I20260928 21:54:27.203395 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/minicluster-data/ts-0/wal/instance
I20260928 21:54:27.203613 2777 external_mini_cluster.cc:1428] Setting key 061b74a2617d667a6b3d1a7160d92339
I20260928 21:54:27.203600 8537 heartbeater.cc:347] Connected to a master server at 127.2.182.126:41921
I20260928 21:54:27.203684 8537 heartbeater.cc:464] Registering TS with master...
I20260928 21:54:27.203883 8537 heartbeater.cc:511] Master 127.2.182.126:41921 requested a full tablet report, sending...
I20260928 21:54:27.204439 8346 ts_manager.cc:194] Registered new tserver with Master: cec5cdc1cb7b43fc83f3c6530cd0841b (127.2.182.65:34953)
I20260928 21:54:27.205365 8346 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.65@KRBTEST.COM'} at 127.2.182.65:56253
WARNING: no policy specified for kudu/127.2.182.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab.
Entry for principal kudu/127.2.182.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab.
I20260928 21:54:27.231122 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.66:0
--local_ip_for_outbound_sockets=127.2.182.66
--webserver_interface=127.2.182.66
--webserver_port=0
--tserver_master_addrs=127.2.182.126:41921
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab
--principal=kudu/127.2.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:37835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:54:27.312417 8542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:54:27.312623 8542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:54:27.312649 8542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:54:27.313983 8542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:54:27.314039 8542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:54:27.314067 8542 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.66
I20260928 21:54:27.315975 8542 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:37835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab
--principal=kudu/127.2.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.182.66
--webserver_port=0
--tserver_master_addrs=127.2.182.126:41921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8542
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.66
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:54:27.316231 8542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:54:27.316468 8542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:54:27.321643 8548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:27.321831 8550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:27.321681 8542 server_base.cc:1034] running on GCE node
W20260928 21:54:27.321637 8547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:27.322115 8542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:27.322329 8542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:27.323462 8542 hybrid_clock.cc:648] HybridClock initialized: now 1790632467323453 us; error 33 us; skew 500 ppm
Sep 28 21:54:27 dist-test-slave-k6lz krb5kdc[8301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632467, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:54:27.326130 8542 init.cc:377] Logged in from keytab as kudu/127.2.182.66@KRBTEST.COM (short username kudu)
I20260928 21:54:27.326596 8542 webserver.cc:460] Webserver started at http://127.2.182.66:46307/ using document root <none> and password file <none>
I20260928 21:54:27.326802 8542 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:27.326850 8542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:27.326970 8542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:54:27.328096 8542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/minicluster-data/ts-1/data/instance:
uuid: "7559fc1256e349b0899b6f5c6973ecd6"
format_stamp: "Formatted at 2026-09-28 21:54:27 on dist-test-slave-k6lz"
server_key: "eadbd647601c644de76399fdec70eeb1"
server_key_iv: "4810c71260a6659b131a664a7fd6893d"
server_key_version: "encryptionkey@0"
I20260928 21:54:27.328429 8542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/minicluster-data/ts-1/wal/instance:
uuid: "7559fc1256e349b0899b6f5c6973ecd6"
format_stamp: "Formatted at 2026-09-28 21:54:27 on dist-test-slave-k6lz"
server_key: "eadbd647601c644de76399fdec70eeb1"
server_key_iv: "4810c71260a6659b131a664a7fd6893d"
server_key_version: "encryptionkey@0"
I20260928 21:54:27.329718 8542 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:54:27.330557 8557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:27.330766 8542 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:54:27.330862 8542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/minicluster-data/ts-1/wal
uuid: "7559fc1256e349b0899b6f5c6973ecd6"
format_stamp: "Formatted at 2026-09-28 21:54:27 on dist-test-slave-k6lz"
server_key: "eadbd647601c644de76399fdec70eeb1"
server_key_iv: "4810c71260a6659b131a664a7fd6893d"
server_key_version: "encryptionkey@0"
I20260928 21:54:27.330935 8542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:27.342579 8542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:54:27.344940 8542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:54:27.345089 8542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:54:27.345320 8542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:54:27.345631 8542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:54:27.345667 8542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:27.345697 8542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:54:27.345721 8542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:27.351655 8542 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.66:44115
I20260928 21:54:27.351701 8670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.66:44115 every 8 connection(s)
I20260928 21:54:27.352025 8542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
I20260928 21:54:27.356022 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 8542
I20260928 21:54:27.356127 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/minicluster-data/ts-1/wal/instance
I20260928 21:54:27.356284 2777 external_mini_cluster.cc:1428] Setting key c0f1fc6d4a364e67cd49b3d7c65ac49b
Sep 28 21:54:27 dist-test-slave-k6lz krb5kdc[8301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632467, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.66@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:54:27.362000 8385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:27.352824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.66:38103 (local address 127.2.182.126:41921)
0928 21:54:27.352896 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:27.352898 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:27.352987 (+ 89us) server_negotiation.cc:407] Connection header received
0928 21:54:27.354038 (+ 1051us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:27.354039 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:27.354049 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:27.354098 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:27.354713 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:27.355276 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:27.356409 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:27.356669 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:27.356729 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:27.359121 (+ 2392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:27.359123 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:27.359123 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:27.359131 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:27.360421 (+ 1290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:27.360923 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:27.360924 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:27.360925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:27.360983 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:27.361321 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:27.361322 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:27.361322 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:27.361450 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:27.361520 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:27.361857 (+ 337us) server_negotiation.cc:299] Negotiation successful
0928 21:54:27.361932 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 21:54:27.362257 8673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:27.352704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:41921 (local address 127.2.182.66:38103)
0928 21:54:27.352853 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:27.352897 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:27.353917 (+ 1020us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:27.354164 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:27.354167 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:27.354424 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:27.354629 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:27.354633 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:27.355391 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:27.355392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:27.356322 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:27.356325 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:27.356745 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:27.356746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:27.356876 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:27.357389 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:27.357399 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:27.359001 (+ 1602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:27.360516 (+ 1515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:27.360525 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:27.360528 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:27.360839 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:27.361047 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:27.361048 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:27.361048 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:27.361186 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:27.361521 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:27.361528 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:27.361706 (+ 178us) client_negotiation.cc:769] Sending connection context
0928 21:54:27.361831 (+ 125us) client_negotiation.cc:240] Negotiation successful
0928 21:54:27.362013 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":65,"threads_started":1}
I20260928 21:54:27.362943 8671 heartbeater.cc:347] Connected to a master server at 127.2.182.126:41921
I20260928 21:54:27.363059 8671 heartbeater.cc:464] Registering TS with master...
I20260928 21:54:27.363273 8671 heartbeater.cc:511] Master 127.2.182.126:41921 requested a full tablet report, sending...
I20260928 21:54:27.363761 8346 ts_manager.cc:194] Registered new tserver with Master: 7559fc1256e349b0899b6f5c6973ecd6 (127.2.182.66:44115)
I20260928 21:54:27.364166 8346 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.66@KRBTEST.COM'} at 127.2.182.66:38103
WARNING: no policy specified for kudu/127.2.182.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab.
Entry for principal kudu/127.2.182.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab.
I20260928 21:54:27.384089 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.67:0
--local_ip_for_outbound_sockets=127.2.182.67
--webserver_interface=127.2.182.67
--webserver_port=0
--tserver_master_addrs=127.2.182.126:41921
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab
--principal=kudu/127.2.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:37835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:54:27.464609 8676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:54:27.464847 8676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:54:27.464870 8676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:54:27.466089 8676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:54:27.466140 8676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:54:27.466168 8676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.67
I20260928 21:54:27.467767 8676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:37835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab
--principal=kudu/127.2.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.182.67
--webserver_port=0
--tserver_master_addrs=127.2.182.126:41921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8676
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.67
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:54:27.468015 8676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:54:27.468235 8676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:54:27.473369 8682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:27.473439 8681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:27.473434 8684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:27.473635 8676 server_base.cc:1034] running on GCE node
I20260928 21:54:27.473886 8676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:27.474114 8676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:27.475313 8676 hybrid_clock.cc:648] HybridClock initialized: now 1790632467475295 us; error 35 us; skew 500 ppm
Sep 28 21:54:27 dist-test-slave-k6lz krb5kdc[8301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632467, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:54:27.478022 8676 init.cc:377] Logged in from keytab as kudu/127.2.182.67@KRBTEST.COM (short username kudu)
I20260928 21:54:27.478503 8676 webserver.cc:460] Webserver started at http://127.2.182.67:37739/ using document root <none> and password file <none>
I20260928 21:54:27.478730 8676 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:27.478778 8676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:27.478902 8676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:54:27.479997 8676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/minicluster-data/ts-2/data/instance:
uuid: "ad409a89e3714f05afb2de361d432311"
format_stamp: "Formatted at 2026-09-28 21:54:27 on dist-test-slave-k6lz"
server_key: "c06bb2801ca49b0486ffdd66b5916361"
server_key_iv: "f4cee3c2bf602bcac24179e426a51b9f"
server_key_version: "encryptionkey@0"
I20260928 21:54:27.480319 8676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/minicluster-data/ts-2/wal/instance:
uuid: "ad409a89e3714f05afb2de361d432311"
format_stamp: "Formatted at 2026-09-28 21:54:27 on dist-test-slave-k6lz"
server_key: "c06bb2801ca49b0486ffdd66b5916361"
server_key_iv: "f4cee3c2bf602bcac24179e426a51b9f"
server_key_version: "encryptionkey@0"
I20260928 21:54:27.481623 8676 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 21:54:27.482389 8691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:27.482576 8676 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:54:27.482673 8676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/minicluster-data/ts-2/wal
uuid: "ad409a89e3714f05afb2de361d432311"
format_stamp: "Formatted at 2026-09-28 21:54:27 on dist-test-slave-k6lz"
server_key: "c06bb2801ca49b0486ffdd66b5916361"
server_key_iv: "f4cee3c2bf602bcac24179e426a51b9f"
server_key_version: "encryptionkey@0"
I20260928 21:54:27.482741 8676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:27.498216 8676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:54:27.500433 8676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:54:27.500579 8676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:54:27.500806 8676 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:54:27.501170 8676 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:54:27.501206 8676 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:27.501236 8676 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:54:27.501259 8676 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:27.506686 8676 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.67:45133
I20260928 21:54:27.506729 8804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.67:45133 every 8 connection(s)
I20260928 21:54:27.507056 8676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
I20260928 21:54:27.509086 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 8676
I20260928 21:54:27.509162 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/minicluster-data/ts-2/wal/instance
I20260928 21:54:27.509327 2777 external_mini_cluster.cc:1428] Setting key ea4198aa368eb12eacd5f74c9fbb494b
Sep 28 21:54:27 dist-test-slave-k6lz krb5kdc[8301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632467, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.67@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:54:27.516360 8385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:27.507800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.67:33337 (local address 127.2.182.126:41921)
0928 21:54:27.507861 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:27.507863 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:27.508015 (+ 152us) server_negotiation.cc:407] Connection header received
0928 21:54:27.508964 (+ 949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:27.508966 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:27.508979 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:27.509066 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:27.509784 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:27.510384 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:27.511323 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:27.511579 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:27.511633 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:27.513970 (+ 2337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:27.513971 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:27.513972 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:27.513979 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:27.515065 (+ 1086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:27.515512 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:27.515512 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:27.515512 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:27.515560 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:27.515759 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:27.515759 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:27.515759 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:27.515947 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:27.515999 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:27.516223 (+ 224us) server_negotiation.cc:299] Negotiation successful
0928 21:54:27.516288 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 21:54:27.516633 8807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:27.507750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:41921 (local address 127.2.182.67:33337)
0928 21:54:27.507893 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:27.507936 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:27.508855 (+ 919us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:27.509120 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:27.509124 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:27.509453 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:27.509705 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:27.509711 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:27.510459 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:27.510460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:27.511256 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:27.511258 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:27.511642 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:27.511643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:27.511752 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:27.512413 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:27.512421 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:27.513865 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:27.515147 (+ 1282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:27.515153 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:27.515155 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:27.515450 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:27.515609 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:27.515609 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:27.515610 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:27.515708 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:27.515998 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:27.516002 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:27.516122 (+ 120us) client_negotiation.cc:769] Sending connection context
0928 21:54:27.516278 (+ 156us) client_negotiation.cc:240] Negotiation successful
0928 21:54:27.516413 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":75,"threads_started":1}
I20260928 21:54:27.517129 8805 heartbeater.cc:347] Connected to a master server at 127.2.182.126:41921
I20260928 21:54:27.517266 8805 heartbeater.cc:464] Registering TS with master...
I20260928 21:54:27.517448 8805 heartbeater.cc:511] Master 127.2.182.126:41921 requested a full tablet report, sending...
I20260928 21:54:27.517797 8346 ts_manager.cc:194] Registered new tserver with Master: ad409a89e3714f05afb2de361d432311 (127.2.182.67:45133)
I20260928 21:54:27.518195 8346 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.67@KRBTEST.COM'} at 127.2.182.67:33337
I20260928 21:54:27.520934 2777 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/krb5kdc/test-admin.keytab.
Sep 28 21:54:27 dist-test-slave-k6lz krb5kdc[8301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:54:27.536341 2777 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260928 21:54:27.539165 8812 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:54:27.537756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47460 (local address 127.2.182.65:34953)
0928 21:54:27.538220 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:27.538228 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:27.538247 (+ 19us) server_negotiation.cc:407] Connection header received
0928 21:54:27.538284 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:27.538287 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:27.538306 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:27.538403 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:27.539065 (+ 662us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47460: BlockingRecv error: recv got EOF from 127.0.0.1:47460 (error 108)
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":235,"threads_started":1}
W20260928 21:54:27.539196 2777 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/krb5kdc/krb5cc' of type FILE
Sep 28 21:54:27 dist-test-slave-k6lz krb5kdc[8301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:54:27.540298 2777 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 21:54:27 dist-test-slave-k6lz krb5kdc[8301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:54:27.541167 2777 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 21:54:27 dist-test-slave-k6lz krb5kdc[8301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.182.65@KRBTEST.COM
I20260928 21:54:27.549733 8812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:27.541820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47464 (local address 127.2.182.65:34953)
0928 21:54:27.541899 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:27.541901 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:27.541938 (+ 37us) server_negotiation.cc:407] Connection header received
0928 21:54:27.542002 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:27.542003 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:27.542016 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:27.542072 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:27.542506 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:27.543268 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:27.544096 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:27.544437 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:27.544497 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:27.546611 (+ 2114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:27.546620 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:27.546620 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:27.546628 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:27.548462 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:27.548851 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:27.548853 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:27.548854 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:27.548913 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:27.549129 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:27.549129 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:27.549130 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:27.549271 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:27.549379 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:27.549535 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 21:54:27.549621 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260928 21:54:27.552081 8812 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:54:27.551470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47470 (local address 127.2.182.65:34953)
0928 21:54:27.551525 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:27.551528 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:27.551650 (+ 122us) server_negotiation.cc:407] Connection header received
0928 21:54:27.551693 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:27.551694 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:27.551704 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:27.551745 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:27.552031 (+ 286us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47470: BlockingRecv error: recv got EOF from 127.0.0.1:47470 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260928 21:54:27.552182 2777 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/krb5kdc/krb5cc' of type FILE
Sep 28 21:54:27 dist-test-slave-k6lz krb5kdc[8301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:54:27.553260 2777 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 21:54:27 dist-test-slave-k6lz krb5kdc[8301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:54:27.554210 2777 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 21:54:27 dist-test-slave-k6lz krb5kdc[8301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.182.65@KRBTEST.COM
I20260928 21:54:27.562515 8812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:27.554794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47486 (local address 127.2.182.65:34953)
0928 21:54:27.554901 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:27.554902 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:27.554910 (+ 8us) server_negotiation.cc:407] Connection header received
0928 21:54:27.554929 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:27.554930 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:27.554940 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:27.554988 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:27.555560 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:27.556140 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:27.556990 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:27.557202 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:27.557271 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:27.559606 (+ 2335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:27.559607 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:27.559607 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:27.559615 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:27.561316 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:27.561670 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:27.561671 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:27.561672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:27.561811 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:27.562067 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:27.562068 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:27.562069 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:27.562183 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:27.562275 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:27.562376 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 21:54:27.562445 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
W20260928 21:54:27.564226 8812 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:54:27.563713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47494 (local address 127.2.182.65:34953)
0928 21:54:27.563869 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:27.563871 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:27.563878 (+ 7us) server_negotiation.cc:407] Connection header received
0928 21:54:27.563894 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:27.563894 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:27.563903 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:27.563946 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:27.564178 (+ 232us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47494: BlockingRecv error: recv got EOF from 127.0.0.1:47494 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260928 21:54:27.564330 2777 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632440055392-2777-0/krb5kdc/krb5cc' of type FILE
Sep 28 21:54:27 dist-test-slave-k6lz krb5kdc[8301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:54:27.565402 2777 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 21:54:27 dist-test-slave-k6lz krb5kdc[8301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:54:27.566169 2777 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 21:54:27 dist-test-slave-k6lz krb5kdc[8301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.182.65@KRBTEST.COM
I20260928 21:54:27.573196 8812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:27.566693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47510 (local address 127.2.182.65:34953)
0928 21:54:27.566795 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:27.566797 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:27.566833 (+ 36us) server_negotiation.cc:407] Connection header received
0928 21:54:27.566887 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:27.566887 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:27.566895 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:27.566938 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:27.567293 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:27.567840 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:27.568614 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:27.568801 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:27.568925 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:27.570916 (+ 1991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:27.570917 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:27.570917 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:27.570924 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:27.572280 (+ 1356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:27.572630 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:27.572631 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:27.572632 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:27.572684 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:27.572833 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:27.572834 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:27.572834 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:27.572924 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:27.572991 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:27.573068 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 21:54:27.573133 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260928 21:54:27.574066 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 8408
I20260928 21:54:27.578035 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 8542
I20260928 21:54:27.581882 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 8676
I20260928 21:54:27.585556 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 8314
2026-09-28T21:54:27Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (5001 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:54:27 dist-test-slave-k6lz krb5kdc[8825](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:54:27 dist-test-slave-k6lz krb5kdc[8825](info): set up 2 sockets
Sep 28 21:54:27 dist-test-slave-k6lz krb5kdc[8825](info): commencing operation
krb5kdc: starting...
W20260928 21:54:28.555514 2777 mini_kdc.cc:121] Time spent starting KDC: real 0.951s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Sep 28 21:54:28 dist-test-slave-k6lz krb5kdc[8825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:54:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:54:28Z Disabled control of system clock
WARNING: no policy specified for oryx/127.2.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/krb5kdc/oryx_127.2.182.126.keytab.
Entry for principal oryx/127.2.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/krb5kdc/oryx_127.2.182.126.keytab.
I20260928 21:54:28.679955 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:43551
--webserver_interface=127.2.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/krb5kdc/oryx_127.2.182.126.keytab
--principal=oryx/127.2.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:45923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.126:43551
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:54:28.761914 8838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:54:28.762151 8838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:54:28.762185 8838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:54:28.763489 8838 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:54:28.763538 8838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:54:28.763557 8838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:54:28.763574 8838 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:54:28.763590 8838 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:54:28.765132 8838 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:45923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.126:43551
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/krb5kdc/oryx_127.2.182.126.keytab
--principal=oryx/127.2.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:43551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:54:28.765383 8838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:54:28.765597 8838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:54:28.770392 8846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:28.770434 8844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:28.770457 8838 server_base.cc:1034] running on GCE node
W20260928 21:54:28.770392 8843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:28.770884 8838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:28.771143 8838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:28.772397 8838 hybrid_clock.cc:648] HybridClock initialized: now 1790632468772371 us; error 38 us; skew 500 ppm
Sep 28 21:54:28 dist-test-slave-k6lz krb5kdc[8825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632468, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:54:28.775032 8838 init.cc:377] Logged in from keytab as oryx/127.2.182.126@KRBTEST.COM (short username oryx)
I20260928 21:54:28.775494 8838 webserver.cc:460] Webserver started at http://127.2.182.126:45001/ using document root <none> and password file <none>
I20260928 21:54:28.775674 8838 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:28.775712 8838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:28.775810 8838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:54:28.776830 8838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/master-0/data/instance:
uuid: "d22a18a4a71545768146dab94c9a04a1"
format_stamp: "Formatted at 2026-09-28 21:54:28 on dist-test-slave-k6lz"
server_key: "fcb41711996b0087d202e34d1b59a2cb"
server_key_iv: "5688e38dfc68367f2510a146d61f5578"
server_key_version: "encryptionkey@0"
I20260928 21:54:28.777124 8838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/master-0/wal/instance:
uuid: "d22a18a4a71545768146dab94c9a04a1"
format_stamp: "Formatted at 2026-09-28 21:54:28 on dist-test-slave-k6lz"
server_key: "fcb41711996b0087d202e34d1b59a2cb"
server_key_iv: "5688e38dfc68367f2510a146d61f5578"
server_key_version: "encryptionkey@0"
I20260928 21:54:28.778362 8838 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 21:54:28.779132 8853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:28.779367 8838 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:54:28.779462 8838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/master-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/master-0/wal
uuid: "d22a18a4a71545768146dab94c9a04a1"
format_stamp: "Formatted at 2026-09-28 21:54:28 on dist-test-slave-k6lz"
server_key: "fcb41711996b0087d202e34d1b59a2cb"
server_key_iv: "5688e38dfc68367f2510a146d61f5578"
server_key_version: "encryptionkey@0"
I20260928 21:54:28.779529 8838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:28.802488 8838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:54:28.805011 8838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:54:28.805156 8838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:54:28.809291 8838 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.126:43551
I20260928 21:54:28.809343 8905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.126:43551 every 8 connection(s)
I20260928 21:54:28.809669 8838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
I20260928 21:54:28.810657 8906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:54:28.813062 8906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d22a18a4a71545768146dab94c9a04a1: Bootstrap starting.
I20260928 21:54:28.813612 8906 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d22a18a4a71545768146dab94c9a04a1: Neither blocks nor log segments found. Creating new log.
I20260928 21:54:28.813856 8906 log.cc:824] T 00000000000000000000000000000000 P d22a18a4a71545768146dab94c9a04a1: Log is configured to *not* fsync() on all Append() calls
I20260928 21:54:28.814586 8906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d22a18a4a71545768146dab94c9a04a1: No bootstrap required, opened a new log
I20260928 21:54:28.815320 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 8838
I20260928 21:54:28.815447 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/master-0/wal/instance
I20260928 21:54:28.815632 2777 external_mini_cluster.cc:1428] Setting key d69e3d3bb3412aadf828c967317388e1
I20260928 21:54:28.816628 8906 raft_consensus.cc:348] T 00000000000000000000000000000000 P d22a18a4a71545768146dab94c9a04a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d22a18a4a71545768146dab94c9a04a1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43551 } }
I20260928 21:54:28.816783 8906 raft_consensus.cc:374] T 00000000000000000000000000000000 P d22a18a4a71545768146dab94c9a04a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:54:28.816819 8906 raft_consensus.cc:729] T 00000000000000000000000000000000 P d22a18a4a71545768146dab94c9a04a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d22a18a4a71545768146dab94c9a04a1, State: Initialized, Role: FOLLOWER
I20260928 21:54:28.816972 8906 consensus_queue.cc:260] T 00000000000000000000000000000000 P d22a18a4a71545768146dab94c9a04a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d22a18a4a71545768146dab94c9a04a1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43551 } }
I20260928 21:54:28.817057 8906 raft_consensus.cc:388] T 00000000000000000000000000000000 P d22a18a4a71545768146dab94c9a04a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:54:28.817087 8906 raft_consensus.cc:482] T 00000000000000000000000000000000 P d22a18a4a71545768146dab94c9a04a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:54:28.817131 8906 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d22a18a4a71545768146dab94c9a04a1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:54:28.818075 8906 raft_consensus.cc:504] T 00000000000000000000000000000000 P d22a18a4a71545768146dab94c9a04a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d22a18a4a71545768146dab94c9a04a1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43551 } }
I20260928 21:54:28.818167 8906 leader_election.cc:302] T 00000000000000000000000000000000 P d22a18a4a71545768146dab94c9a04a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d22a18a4a71545768146dab94c9a04a1; no voters:
I20260928 21:54:28.818321 8906 leader_election.cc:288] T 00000000000000000000000000000000 P d22a18a4a71545768146dab94c9a04a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:54:28.818364 8911 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d22a18a4a71545768146dab94c9a04a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:54:28.818519 8911 raft_consensus.cc:686] T 00000000000000000000000000000000 P d22a18a4a71545768146dab94c9a04a1 [term 1 LEADER]: Becoming Leader. State: Replica: d22a18a4a71545768146dab94c9a04a1, State: Running, Role: LEADER
I20260928 21:54:28.818580 8906 sys_catalog.cc:564] T 00000000000000000000000000000000 P d22a18a4a71545768146dab94c9a04a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:54:28.818627 8911 consensus_queue.cc:237] T 00000000000000000000000000000000 P d22a18a4a71545768146dab94c9a04a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d22a18a4a71545768146dab94c9a04a1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43551 } }
I20260928 21:54:28.819149 8912 sys_catalog.cc:455] T 00000000000000000000000000000000 P d22a18a4a71545768146dab94c9a04a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d22a18a4a71545768146dab94c9a04a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d22a18a4a71545768146dab94c9a04a1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43551 } } }
I20260928 21:54:28.819244 8912 sys_catalog.cc:458] T 00000000000000000000000000000000 P d22a18a4a71545768146dab94c9a04a1 [sys.catalog]: This master's current role is: LEADER
I20260928 21:54:28.819130 8913 sys_catalog.cc:455] T 00000000000000000000000000000000 P d22a18a4a71545768146dab94c9a04a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d22a18a4a71545768146dab94c9a04a1. Latest consensus state: current_term: 1 leader_uuid: "d22a18a4a71545768146dab94c9a04a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d22a18a4a71545768146dab94c9a04a1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43551 } } }
I20260928 21:54:28.819600 8913 sys_catalog.cc:458] T 00000000000000000000000000000000 P d22a18a4a71545768146dab94c9a04a1 [sys.catalog]: This master's current role is: LEADER
I20260928 21:54:28.819451 8920 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:54:28.820358 8920 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:54:28.821990 8920 catalog_manager.cc:1347] Generated new cluster ID: 088e5e211b3d41aa93ae71942e3a4aab
I20260928 21:54:28.822036 8920 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 21:54:28 dist-test-slave-k6lz krb5kdc[8825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM
I20260928 21:54:28.826535 8910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:28.816175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46326 (local address 127.2.182.126:43551)
0928 21:54:28.816339 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:28.816347 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:28.816383 (+ 36us) server_negotiation.cc:407] Connection header received
0928 21:54:28.816586 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:28.816591 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:28.816635 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:28.816742 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:28.817470 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:28.818534 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:28.819581 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:28.820160 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:28.820310 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:28.822985 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:28.822990 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:28.822991 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:28.823002 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:28.824905 (+ 1903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:28.825340 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:28.825344 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:28.825344 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:28.825464 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:28.825664 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:28.825664 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:28.825665 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:28.825947 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:28.826085 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:28.826198 (+ 113us) server_negotiation.cc:299] Negotiation successful
0928 21:54:28.826337 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":71,"threads_started":1}
I20260928 21:54:28.841866 8920 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:54:28.842630 8920 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:54:28.848588 8920 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d22a18a4a71545768146dab94c9a04a1: Generated new TSK 0
I20260928 21:54:28.848791 8920 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.2.182.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.182.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.182.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/krb5kdc/oryx_127.2.182.65.keytab.
Entry for principal oryx/127.2.182.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/krb5kdc/oryx_127.2.182.65.keytab.
I20260928 21:54:28.883859 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.65:0
--local_ip_for_outbound_sockets=127.2.182.65
--webserver_interface=127.2.182.65
--webserver_port=0
--tserver_master_addrs=127.2.182.126:43551
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/krb5kdc/oryx_127.2.182.65.keytab
--principal=oryx/127.2.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:45923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:54:28.985889 8932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:54:28.986114 8932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:54:28.986138 8932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:54:28.987408 8932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:54:28.987458 8932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:54:28.987485 8932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.65
I20260928 21:54:28.989053 8932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:45923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/krb5kdc/oryx_127.2.182.65.keytab
--principal=oryx/127.2.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.182.65
--webserver_port=0
--tserver_master_addrs=127.2.182.126:43551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.65
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:54:28.989302 8932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:54:28.989506 8932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:54:28.994738 8940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:28.994719 8937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:28.994750 8938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:28.994769 8932 server_base.cc:1034] running on GCE node
I20260928 21:54:28.995205 8932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:28.995488 8932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:28.996640 8932 hybrid_clock.cc:648] HybridClock initialized: now 1790632468996620 us; error 35 us; skew 500 ppm
Sep 28 21:54:28 dist-test-slave-k6lz krb5kdc[8825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632468, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:54:28.999176 8932 init.cc:377] Logged in from keytab as oryx/127.2.182.65@KRBTEST.COM (short username oryx)
I20260928 21:54:28.999722 8932 webserver.cc:460] Webserver started at http://127.2.182.65:35807/ using document root <none> and password file <none>
I20260928 21:54:28.999957 8932 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:29.000128 8932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:29.000301 8932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:54:29.001430 8932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-0/data/instance:
uuid: "4438ab182bef4715bd7a150fab1403bd"
format_stamp: "Formatted at 2026-09-28 21:54:28 on dist-test-slave-k6lz"
server_key: "ea0a718ca73f9257b2b72d1c45d1cc86"
server_key_iv: "eefcb0cec937eda91eee238205e9eded"
server_key_version: "encryptionkey@0"
I20260928 21:54:29.001760 8932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-0/wal/instance:
uuid: "4438ab182bef4715bd7a150fab1403bd"
format_stamp: "Formatted at 2026-09-28 21:54:28 on dist-test-slave-k6lz"
server_key: "ea0a718ca73f9257b2b72d1c45d1cc86"
server_key_iv: "eefcb0cec937eda91eee238205e9eded"
server_key_version: "encryptionkey@0"
I20260928 21:54:29.003126 8932 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 21:54:29.004021 8947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:29.004266 8932 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:54:29.004405 8932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-0/wal
uuid: "4438ab182bef4715bd7a150fab1403bd"
format_stamp: "Formatted at 2026-09-28 21:54:28 on dist-test-slave-k6lz"
server_key: "ea0a718ca73f9257b2b72d1c45d1cc86"
server_key_iv: "eefcb0cec937eda91eee238205e9eded"
server_key_version: "encryptionkey@0"
I20260928 21:54:29.004509 8932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:29.013164 8932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:54:29.015321 8932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:54:29.015503 8932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:54:29.015733 8932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:54:29.016031 8932 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:54:29.016064 8932 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:29.016093 8932 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:54:29.016116 8932 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:29.021799 8932 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.65:39731
I20260928 21:54:29.021843 9060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.65:39731 every 8 connection(s)
I20260928 21:54:29.022145 8932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
Sep 28 21:54:29 dist-test-slave-k6lz krb5kdc[8825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632468, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.65@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM
I20260928 21:54:29.029981 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 8932
I20260928 21:54:29.030067 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-0/wal/instance
I20260928 21:54:29.030256 2777 external_mini_cluster.cc:1428] Setting key c0205ba68d15b87d989d07366ffbe6ac
I20260928 21:54:29.031607 8910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:29.023009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.65:45557 (local address 127.2.182.126:43551)
0928 21:54:29.023069 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:29.023071 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:29.023190 (+ 119us) server_negotiation.cc:407] Connection header received
0928 21:54:29.023718 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:29.023719 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:29.023728 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:29.023789 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:29.024713 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:29.025293 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:29.026313 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:29.026528 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:29.026585 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:29.028819 (+ 2234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:29.028821 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:29.028821 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:29.028831 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:29.030155 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:29.030628 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:29.030629 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:29.030630 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:29.030692 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:29.030895 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:29.030896 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:29.030896 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:29.031025 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:29.031086 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:29.031455 (+ 369us) server_negotiation.cc:299] Negotiation successful
0928 21:54:29.031536 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 21:54:29.031829 9063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:29.022898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:43551 (local address 127.2.182.65:45557)
0928 21:54:29.023067 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:29.023109 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:29.023628 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:29.023849 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:29.023852 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:29.024422 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:29.024617 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:29.024622 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:29.025388 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:29.025389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:29.026221 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:29.026223 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:29.026631 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:29.026632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:29.026747 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:29.027288 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:29.027296 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:29.028727 (+ 1431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:29.030224 (+ 1497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:29.030231 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:29.030233 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:29.030547 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:29.030744 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:29.030744 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:29.030745 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:29.030839 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:29.031095 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:29.031101 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:29.031264 (+ 163us) client_negotiation.cc:769] Sending connection context
0928 21:54:29.031444 (+ 180us) client_negotiation.cc:240] Negotiation successful
0928 21:54:29.031615 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":68,"threads_started":1}
I20260928 21:54:29.032349 9061 heartbeater.cc:347] Connected to a master server at 127.2.182.126:43551
I20260928 21:54:29.032462 9061 heartbeater.cc:464] Registering TS with master...
I20260928 21:54:29.032661 9061 heartbeater.cc:511] Master 127.2.182.126:43551 requested a full tablet report, sending...
I20260928 21:54:29.033201 8870 ts_manager.cc:194] Registered new tserver with Master: 4438ab182bef4715bd7a150fab1403bd (127.2.182.65:39731)
I20260928 21:54:29.033886 8870 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.182.65@KRBTEST.COM'} at 127.2.182.65:45557
WARNING: no policy specified for oryx/127.2.182.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.182.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.182.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/krb5kdc/oryx_127.2.182.66.keytab.
Entry for principal oryx/127.2.182.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/krb5kdc/oryx_127.2.182.66.keytab.
I20260928 21:54:29.058362 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.66:0
--local_ip_for_outbound_sockets=127.2.182.66
--webserver_interface=127.2.182.66
--webserver_port=0
--tserver_master_addrs=127.2.182.126:43551
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/krb5kdc/oryx_127.2.182.66.keytab
--principal=oryx/127.2.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:45923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:54:29.138891 9066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:54:29.139098 9066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:54:29.139122 9066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:54:29.140261 9066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:54:29.140307 9066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:54:29.140336 9066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.66
I20260928 21:54:29.141767 9066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:45923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/krb5kdc/oryx_127.2.182.66.keytab
--principal=oryx/127.2.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.182.66
--webserver_port=0
--tserver_master_addrs=127.2.182.126:43551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9066
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.66
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:54:29.141980 9066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:54:29.142168 9066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:54:29.147055 9074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:29.147135 9071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:29.147166 9066 server_base.cc:1034] running on GCE node
W20260928 21:54:29.147078 9072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:29.147495 9066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:29.147706 9066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:29.148873 9066 hybrid_clock.cc:648] HybridClock initialized: now 1790632469148859 us; error 27 us; skew 500 ppm
Sep 28 21:54:29 dist-test-slave-k6lz krb5kdc[8825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632469, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:54:29.151427 9066 init.cc:377] Logged in from keytab as oryx/127.2.182.66@KRBTEST.COM (short username oryx)
I20260928 21:54:29.151853 9066 webserver.cc:460] Webserver started at http://127.2.182.66:34861/ using document root <none> and password file <none>
I20260928 21:54:29.152061 9066 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:29.152109 9066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:29.152232 9066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:54:29.153282 9066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-1/data/instance:
uuid: "d6cf3fea18684e48bc758ab6bf07563e"
format_stamp: "Formatted at 2026-09-28 21:54:29 on dist-test-slave-k6lz"
server_key: "f0b8c51e5d3fc40804660103806ae165"
server_key_iv: "893f03ce1c74ec9a1f609459ef8cfefe"
server_key_version: "encryptionkey@0"
I20260928 21:54:29.153621 9066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-1/wal/instance:
uuid: "d6cf3fea18684e48bc758ab6bf07563e"
format_stamp: "Formatted at 2026-09-28 21:54:29 on dist-test-slave-k6lz"
server_key: "f0b8c51e5d3fc40804660103806ae165"
server_key_iv: "893f03ce1c74ec9a1f609459ef8cfefe"
server_key_version: "encryptionkey@0"
I20260928 21:54:29.155045 9066 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:54:29.155984 9081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:29.156217 9066 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:54:29.156311 9066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-1/wal
uuid: "d6cf3fea18684e48bc758ab6bf07563e"
format_stamp: "Formatted at 2026-09-28 21:54:29 on dist-test-slave-k6lz"
server_key: "f0b8c51e5d3fc40804660103806ae165"
server_key_iv: "893f03ce1c74ec9a1f609459ef8cfefe"
server_key_version: "encryptionkey@0"
I20260928 21:54:29.156391 9066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:29.176095 9066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:54:29.178282 9066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:54:29.178423 9066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:54:29.178655 9066 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:54:29.178964 9066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:54:29.178997 9066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:29.179026 9066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:54:29.179052 9066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:29.184715 9066 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.66:33051
I20260928 21:54:29.184763 9194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.66:33051 every 8 connection(s)
I20260928 21:54:29.185081 9066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
Sep 28 21:54:29 dist-test-slave-k6lz krb5kdc[8825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632469, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.66@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM
I20260928 21:54:29.193457 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 9066
I20260928 21:54:29.193549 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-1/wal/instance
I20260928 21:54:29.193755 2777 external_mini_cluster.cc:1428] Setting key da92ef347715ee222e4c2b29aa40cb4f
I20260928 21:54:29.194648 8910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:29.185887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.66:38999 (local address 127.2.182.126:43551)
0928 21:54:29.185956 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:29.185958 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:29.186124 (+ 166us) server_negotiation.cc:407] Connection header received
0928 21:54:29.186754 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:29.186756 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:29.186775 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:29.186835 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:29.187893 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:29.188405 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:29.189459 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:29.189675 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:29.189735 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:29.191893 (+ 2158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:29.191895 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:29.191895 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:29.191902 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:29.192997 (+ 1095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:29.193477 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:29.193478 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:29.193478 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:29.193530 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:29.193991 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:29.193992 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:29.193992 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:29.194112 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:29.194163 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:29.194471 (+ 308us) server_negotiation.cc:299] Negotiation successful
0928 21:54:29.194560 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 21:54:29.194813 9197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:29.185847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:43551 (local address 127.2.182.66:38999)
0928 21:54:29.186007 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:29.186045 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:29.186551 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:29.186952 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:29.186955 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:29.187592 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:29.187788 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:29.187793 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:29.188507 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:29.188508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:29.189361 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:29.189364 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:29.189754 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:29.189755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:29.189861 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:29.190362 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:29.190370 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:29.191809 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:29.193108 (+ 1299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:29.193114 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:29.193116 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:29.193402 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:29.193630 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:29.193631 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:29.193631 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:29.193880 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:29.194169 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:29.194175 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:29.194342 (+ 167us) client_negotiation.cc:769] Sending connection context
0928 21:54:29.194437 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 21:54:29.194592 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":78,"threads_started":1}
I20260928 21:54:29.195289 9195 heartbeater.cc:347] Connected to a master server at 127.2.182.126:43551
I20260928 21:54:29.195420 9195 heartbeater.cc:464] Registering TS with master...
I20260928 21:54:29.195607 9195 heartbeater.cc:511] Master 127.2.182.126:43551 requested a full tablet report, sending...
I20260928 21:54:29.196007 8870 ts_manager.cc:194] Registered new tserver with Master: d6cf3fea18684e48bc758ab6bf07563e (127.2.182.66:33051)
I20260928 21:54:29.196501 8870 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.182.66@KRBTEST.COM'} at 127.2.182.66:38999
WARNING: no policy specified for oryx/127.2.182.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.182.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.182.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/krb5kdc/oryx_127.2.182.67.keytab.
Entry for principal oryx/127.2.182.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/krb5kdc/oryx_127.2.182.67.keytab.
I20260928 21:54:29.221938 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.67:0
--local_ip_for_outbound_sockets=127.2.182.67
--webserver_interface=127.2.182.67
--webserver_port=0
--tserver_master_addrs=127.2.182.126:43551
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/krb5kdc/oryx_127.2.182.67.keytab
--principal=oryx/127.2.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:45923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:54:29.304853 9200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:54:29.305080 9200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:54:29.305117 9200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:54:29.306470 9200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:54:29.306520 9200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:54:29.306548 9200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.67
I20260928 21:54:29.308192 9200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:45923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/krb5kdc/oryx_127.2.182.67.keytab
--principal=oryx/127.2.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.182.67
--webserver_port=0
--tserver_master_addrs=127.2.182.126:43551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.67
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:54:29.308449 9200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:54:29.308671 9200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:54:29.313941 9206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:29.314015 9205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:29.314006 9200 server_base.cc:1034] running on GCE node
W20260928 21:54:29.313941 9208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:29.314435 9200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:29.314685 9200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:29.315857 9200 hybrid_clock.cc:648] HybridClock initialized: now 1790632469315804 us; error 69 us; skew 500 ppm
Sep 28 21:54:29 dist-test-slave-k6lz krb5kdc[8825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632469, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:54:29.318518 9200 init.cc:377] Logged in from keytab as oryx/127.2.182.67@KRBTEST.COM (short username oryx)
I20260928 21:54:29.319015 9200 webserver.cc:460] Webserver started at http://127.2.182.67:43437/ using document root <none> and password file <none>
I20260928 21:54:29.319294 9200 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:29.319433 9200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:29.319587 9200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:54:29.320655 9200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-2/data/instance:
uuid: "65646a62d4364636adf7ec56574214c0"
format_stamp: "Formatted at 2026-09-28 21:54:29 on dist-test-slave-k6lz"
server_key: "22820945eb66ada5ac565fc871aeb7f5"
server_key_iv: "5023178be24e82bb02cec421d0674fa7"
server_key_version: "encryptionkey@0"
I20260928 21:54:29.320952 9200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-2/wal/instance:
uuid: "65646a62d4364636adf7ec56574214c0"
format_stamp: "Formatted at 2026-09-28 21:54:29 on dist-test-slave-k6lz"
server_key: "22820945eb66ada5ac565fc871aeb7f5"
server_key_iv: "5023178be24e82bb02cec421d0674fa7"
server_key_version: "encryptionkey@0"
I20260928 21:54:29.322201 9200 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:54:29.323139 9215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:29.323411 9200 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:54:29.323496 9200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-2/wal
uuid: "65646a62d4364636adf7ec56574214c0"
format_stamp: "Formatted at 2026-09-28 21:54:29 on dist-test-slave-k6lz"
server_key: "22820945eb66ada5ac565fc871aeb7f5"
server_key_iv: "5023178be24e82bb02cec421d0674fa7"
server_key_version: "encryptionkey@0"
I20260928 21:54:29.323555 9200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:29.355674 9200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:54:29.357760 9200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:54:29.357874 9200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:54:29.358055 9200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:54:29.358349 9200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:54:29.358381 9200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:29.358404 9200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:54:29.358425 9200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:29.363862 9200 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.67:33249
I20260928 21:54:29.363929 9328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.67:33249 every 8 connection(s)
I20260928 21:54:29.364208 9200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
I20260928 21:54:29.367426 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 9200
I20260928 21:54:29.367517 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-2/wal/instance
I20260928 21:54:29.367712 2777 external_mini_cluster.cc:1428] Setting key 08a8236fc14c878f867c75e25b849ddf
Sep 28 21:54:29 dist-test-slave-k6lz krb5kdc[8825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632469, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.67@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM
I20260928 21:54:29.374611 8910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:29.364964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.67:33449 (local address 127.2.182.126:43551)
0928 21:54:29.365027 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:29.365030 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:29.365372 (+ 342us) server_negotiation.cc:407] Connection header received
0928 21:54:29.365807 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:29.365808 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:29.365983 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:29.366104 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:29.367250 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:29.368124 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:29.369283 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:29.369471 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:29.369528 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:29.371637 (+ 2109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:29.371639 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:29.371640 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:29.371646 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:29.373063 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:29.373488 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:29.373489 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:29.373490 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:29.373557 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:29.373924 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:29.373925 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:29.373925 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:29.374060 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:29.374117 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:29.374470 (+ 353us) server_negotiation.cc:299] Negotiation successful
0928 21:54:29.374543 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 21:54:29.374866 9331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:29.364934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:43551 (local address 127.2.182.67:33449)
0928 21:54:29.365132 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:29.365170 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:29.365693 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:29.366215 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:29.366219 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:29.366820 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:29.367104 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:29.367109 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:29.368208 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:29.368209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:29.369179 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:29.369182 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:29.369536 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:29.369536 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:29.369635 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:29.370115 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:29.370124 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:29.371527 (+ 1403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:29.373143 (+ 1616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:29.373150 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:29.373152 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:29.373396 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:29.373648 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:29.373649 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:29.373649 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:29.373846 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:29.374172 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:29.374178 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:29.374326 (+ 148us) client_negotiation.cc:769] Sending connection context
0928 21:54:29.374434 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 21:54:29.374575 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":86,"threads_started":1}
I20260928 21:54:29.375401 9329 heartbeater.cc:347] Connected to a master server at 127.2.182.126:43551
I20260928 21:54:29.375514 9329 heartbeater.cc:464] Registering TS with master...
I20260928 21:54:29.375696 9329 heartbeater.cc:511] Master 127.2.182.126:43551 requested a full tablet report, sending...
I20260928 21:54:29.376107 8870 ts_manager.cc:194] Registered new tserver with Master: 65646a62d4364636adf7ec56574214c0 (127.2.182.67:33249)
I20260928 21:54:29.376566 8870 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.182.67@KRBTEST.COM'} at 127.2.182.67:33449
I20260928 21:54:29.379999 2777 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.2.182.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.182.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.182.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/krb5kdc/oryx_127.2.182.125.keytab.
Entry for principal oryx/127.2.182.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/krb5kdc/oryx_127.2.182.125.keytab.
I20260928 21:54:29.411576 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.125:33661
--webserver_interface=127.2.182.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/krb5kdc/oryx_127.2.182.125.keytab
--principal=oryx/127.2.182.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:45923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.126:43551,127.2.182.125:33661
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:54:29.498776 9334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:54:29.499043 9334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:54:29.499082 9334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:54:29.500432 9334 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:54:29.500497 9334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:54:29.500516 9334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:54:29.500532 9334 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:54:29.500547 9334 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:54:29.502381 9334 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:45923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.126:43551,127.2.182.125:33661
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/krb5kdc/oryx_127.2.182.125.keytab
--principal=oryx/127.2.182.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.125:33661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.182.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9334
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
W20260928 21:54:29.502534 9334 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.2.182.126:43551,127.2.182.125:33661'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260928 21:54:29.502710 9334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:54:29.502985 9334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:54:29.508558 9339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:29.508708 9340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:29.508858 9342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:29.508900 9334 server_base.cc:1034] running on GCE node
I20260928 21:54:29.509146 9334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:29.509496 9334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:29.510664 9334 hybrid_clock.cc:648] HybridClock initialized: now 1790632469510650 us; error 28 us; skew 500 ppm
Sep 28 21:54:29 dist-test-slave-k6lz krb5kdc[8825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632469, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:54:29.513429 9334 init.cc:377] Logged in from keytab as oryx/127.2.182.125@KRBTEST.COM (short username oryx)
I20260928 21:54:29.513876 9334 webserver.cc:460] Webserver started at http://127.2.182.125:33507/ using document root <none> and password file <none>
I20260928 21:54:29.514083 9334 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:29.514140 9334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:29.514250 9334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:54:29.515408 9334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/master-1/data/instance:
uuid: "59256c4c13d94676ac3e3b7e1e51a6a8"
format_stamp: "Formatted at 2026-09-28 21:54:29 on dist-test-slave-k6lz"
server_key: "5e14299801f4267c55111fb489a08437"
server_key_iv: "e88034f8c1790aa359e4c01b23857e23"
server_key_version: "encryptionkey@0"
I20260928 21:54:29.515740 9334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/master-1/wal/instance:
uuid: "59256c4c13d94676ac3e3b7e1e51a6a8"
format_stamp: "Formatted at 2026-09-28 21:54:29 on dist-test-slave-k6lz"
server_key: "5e14299801f4267c55111fb489a08437"
server_key_iv: "e88034f8c1790aa359e4c01b23857e23"
server_key_version: "encryptionkey@0"
I20260928 21:54:29.517089 9334 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:54:29.518036 9349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:29.518431 9334 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:54:29.518682 9334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/master-1/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/master-1/wal
uuid: "59256c4c13d94676ac3e3b7e1e51a6a8"
format_stamp: "Formatted at 2026-09-28 21:54:29 on dist-test-slave-k6lz"
server_key: "5e14299801f4267c55111fb489a08437"
server_key_iv: "e88034f8c1790aa359e4c01b23857e23"
server_key_version: "encryptionkey@0"
I20260928 21:54:29.518937 9334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:29.531558 9334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:54:29.533928 9334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:54:29.534053 9334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:54:29.538165 9334 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.125:33661
I20260928 21:54:29.538216 9401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.125:33661 every 8 connection(s)
I20260928 21:54:29.538533 9334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/master-1/data/info.pb
I20260928 21:54:29.539634 9402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:54:29.541154 9402 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43551 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 21:54:29 dist-test-slave-k6lz krb5kdc[8825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632469, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.125@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM
I20260928 21:54:29.547726 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 9334
I20260928 21:54:29.552481 8910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:29.541942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46336 (local address 127.2.182.126:43551)
0928 21:54:29.542027 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:29.542029 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:29.542282 (+ 253us) server_negotiation.cc:407] Connection header received
0928 21:54:29.542807 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:29.542809 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:29.542822 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:29.542875 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:29.543551 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:29.544119 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:29.545126 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:29.545324 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:29.545382 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:29.548249 (+ 2867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:29.548252 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:29.548253 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:29.548267 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:29.550428 (+ 2161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:29.550990 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:29.550991 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:29.550992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:29.551071 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:29.551628 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:29.551629 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:29.551630 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:29.551830 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:29.551886 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:29.552299 (+ 413us) server_negotiation.cc:299] Negotiation successful
0928 21:54:29.552392 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 21:54:29.552560 9404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:29.541956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:43551 (local address 127.0.0.1:46336)
0928 21:54:29.542160 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:29.542197 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:29.542701 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:29.542949 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:29.542952 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:29.543206 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:29.543462 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:29.543466 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:29.544200 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:29.544202 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:29.545026 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:29.545028 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:29.545401 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:29.545402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:29.545523 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:29.546107 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:29.546116 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:29.548006 (+ 1890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:29.550566 (+ 2560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:29.550574 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:29.550576 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:29.550891 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:29.551232 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:29.551234 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:29.551234 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:29.551547 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:29.551894 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:29.551898 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:29.552058 (+ 160us) client_negotiation.cc:769] Sending connection context
0928 21:54:29.552164 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 21:54:29.552306 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":95,"threads_started":1}
I20260928 21:54:29.553188 9402 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.182.125" port: 33661 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 21:54:29 dist-test-slave-k6lz krb5kdc[8825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.182.125@KRBTEST.COM
Sep 28 21:54:29 dist-test-slave-k6lz krb5kdc[8825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632469, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.125@KRBTEST.COM for oryx/127.2.182.125@KRBTEST.COM
I20260928 21:54:29.557739 9406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:29.548515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36986 (local address 127.2.182.125:33661)
0928 21:54:29.548719 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:29.548725 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:29.548748 (+ 23us) server_negotiation.cc:407] Connection header received
0928 21:54:29.548769 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:29.548771 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:29.548784 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:29.548987 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:29.549537 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:29.550635 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:29.551628 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:29.552013 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:29.552084 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:29.554841 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:29.554849 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:29.554850 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:29.554856 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:29.556596 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:29.556946 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:29.556949 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:29.556949 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:29.557001 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:29.557169 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:29.557169 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:29.557170 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:29.557325 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:29.557485 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:29.557599 (+ 114us) server_negotiation.cc:299] Negotiation successful
0928 21:54:29.557661 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":97,"threads_started":1}
I20260928 21:54:29.560737 9404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:29.553567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.125:33661 (local address 127.0.0.1:36996)
0928 21:54:29.553624 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:29.553636 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:29.553694 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:29.553864 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:29.553866 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:29.554007 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:29.554065 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:29.554066 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:29.554808 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:29.554809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:29.555307 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:29.555309 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:29.555790 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:29.555791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:29.555854 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:29.556247 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:29.556252 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:29.557487 (+ 1235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:29.559575 (+ 2088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:29.559576 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:29.559576 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:29.559729 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:29.559954 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:29.559955 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:29.559955 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:29.560013 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:29.560476 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:29.560477 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:29.560526 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 21:54:29.560599 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 21:54:29.560672 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 21:54:29.560956 9407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:29.553615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36996 (local address 127.2.182.125:33661)
0928 21:54:29.553759 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:29.553760 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:29.553767 (+ 7us) server_negotiation.cc:407] Connection header received
0928 21:54:29.553782 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:29.553783 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:29.553790 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:29.553845 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:29.554163 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:29.554725 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:29.555417 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:29.555727 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:29.555789 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:29.557572 (+ 1783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:29.557574 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:29.557574 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:29.557580 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:29.559503 (+ 1923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:29.559822 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:29.559823 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:29.559824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:29.559890 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:29.560155 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:29.560158 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:29.560158 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:29.560360 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:29.560452 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:29.560680 (+ 228us) server_negotiation.cc:299] Negotiation successful
0928 21:54:29.560755 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":90,"threads_started":1}
I20260928 21:54:29.562196 9402 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59256c4c13d94676ac3e3b7e1e51a6a8: Bootstrap starting.
I20260928 21:54:29.562839 9402 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59256c4c13d94676ac3e3b7e1e51a6a8: Neither blocks nor log segments found. Creating new log.
I20260928 21:54:29.563112 9402 log.cc:824] T 00000000000000000000000000000000 P 59256c4c13d94676ac3e3b7e1e51a6a8: Log is configured to *not* fsync() on all Append() calls
I20260928 21:54:29.564288 9402 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59256c4c13d94676ac3e3b7e1e51a6a8: No bootstrap required, opened a new log
I20260928 21:54:29.566478 9402 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59256c4c13d94676ac3e3b7e1e51a6a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d22a18a4a71545768146dab94c9a04a1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43551 } } peers { permanent_uuid: "59256c4c13d94676ac3e3b7e1e51a6a8" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 33661 } }
I20260928 21:54:29.566646 9402 raft_consensus.cc:374] T 00000000000000000000000000000000 P 59256c4c13d94676ac3e3b7e1e51a6a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:54:29.566680 9402 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59256c4c13d94676ac3e3b7e1e51a6a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59256c4c13d94676ac3e3b7e1e51a6a8, State: Initialized, Role: FOLLOWER
I20260928 21:54:29.566838 9402 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59256c4c13d94676ac3e3b7e1e51a6a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d22a18a4a71545768146dab94c9a04a1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43551 } } peers { permanent_uuid: "59256c4c13d94676ac3e3b7e1e51a6a8" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 33661 } }
I20260928 21:54:29.567153 9402 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59256c4c13d94676ac3e3b7e1e51a6a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:54:29.567212 9409 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59256c4c13d94676ac3e3b7e1e51a6a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d22a18a4a71545768146dab94c9a04a1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43551 } } peers { permanent_uuid: "59256c4c13d94676ac3e3b7e1e51a6a8" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 33661 } } }
I20260928 21:54:29.567417 9409 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59256c4c13d94676ac3e3b7e1e51a6a8 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 21:54:29.569113 9420 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 59256c4c13d94676ac3e3b7e1e51a6a8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:54:29.569193 9420 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:54:29.571247 9334 master_runner.cc:428] Detected that this master 59256c4c13d94676ac3e3b7e1e51a6a8 is joining an existing cluster
I20260928 21:54:29.571321 9334 master_runner.cc:432] Initiating AddMaster RPC to add 127.2.182.125:33661
I20260928 21:54:29.579195 9426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:29.572541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.125:33661 (local address 127.0.0.1:37006)
0928 21:54:29.572689 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:29.572698 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:29.572755 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:29.572897 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:29.572898 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:29.573129 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:29.573196 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:29.573198 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:29.574085 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:29.574085 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:29.574683 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:29.574684 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:29.575041 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:29.575042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:29.575098 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:29.575637 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:29.575642 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:29.576048 (+ 406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:29.577817 (+ 1769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:29.577818 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:29.577819 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:29.578039 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:29.578307 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:29.578308 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:29.578309 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:29.578384 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:29.578805 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:29.578806 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:29.578850 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 21:54:29.578936 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 21:54:29.579067 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":58,"threads_started":1}
I20260928 21:54:29.579439 9407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:29.572629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37006 (local address 127.2.182.125:33661)
0928 21:54:29.572687 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:29.572689 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:29.572777 (+ 88us) server_negotiation.cc:407] Connection header received
0928 21:54:29.572799 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:29.572799 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:29.572808 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:29.572850 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:29.573262 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:29.573995 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:29.574756 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:29.574957 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:29.575011 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:29.576135 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:29.576136 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:29.576136 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:29.576142 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:29.577720 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:29.578122 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:29.578123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:29.578123 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:29.578172 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:29.578465 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:29.578466 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:29.578466 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:29.578601 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:29.578656 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:29.578977 (+ 321us) server_negotiation.cc:299] Negotiation successful
0928 21:54:29.579277 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":141,"server-negotiator.queue_time_us":39}
I20260928 21:54:29.579792 8910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:29.572510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46350 (local address 127.2.182.126:43551)
0928 21:54:29.572554 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:29.572556 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:29.572759 (+ 203us) server_negotiation.cc:407] Connection header received
0928 21:54:29.572863 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:29.572864 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:29.572978 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:29.573035 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:29.573421 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:29.574140 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:29.575193 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:29.575463 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:29.575519 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:29.576359 (+ 840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:29.576360 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:29.576361 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:29.576368 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:29.577971 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:29.578417 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:29.578423 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:29.578423 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:29.578604 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:29.579082 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:29.579084 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:29.579084 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:29.579268 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:29.579323 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:29.579556 (+ 233us) server_negotiation.cc:299] Negotiation successful
0928 21:54:29.579665 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 21:54:29.579794 9427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:29.572478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:43551 (local address 127.0.0.1:46350)
0928 21:54:29.572674 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:29.572686 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:29.572761 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:29.573053 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:29.573054 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:29.573270 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:29.573340 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:29.573342 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:29.574215 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:29.574216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:29.575106 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:29.575109 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:29.575569 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:29.575570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:29.575629 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:29.575978 (+ 349us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:29.575983 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:29.576297 (+ 314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:29.578041 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:29.578042 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:29.578042 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:29.578242 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:29.578864 (+ 622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:29.578866 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:29.578867 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:29.578988 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:29.579406 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:29.579407 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:29.579455 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 21:54:29.579513 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 21:54:29.579680 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":84,"threads_started":1}
Sep 28 21:54:29 dist-test-slave-k6lz krb5kdc[8825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632468, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.126@KRBTEST.COM for oryx/127.2.182.125@KRBTEST.COM
I20260928 21:54:29.593752 9407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:29.583397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37022 (local address 127.2.182.125:33661)
0928 21:54:29.583453 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:29.583456 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:29.583662 (+ 206us) server_negotiation.cc:407] Connection header received
0928 21:54:29.583724 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:29.583724 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:29.583735 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:29.583802 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:29.584255 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:29.585105 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:29.586312 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:29.586627 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:29.586701 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:29.589482 (+ 2781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:29.589483 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:29.589484 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:29.589491 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:29.591582 (+ 2091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:29.592447 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:29.592449 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:29.592449 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:29.592525 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:29.592872 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:29.592872 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:29.592873 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:29.593033 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:29.593110 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:29.593502 (+ 392us) server_negotiation.cc:299] Negotiation successful
0928 21:54:29.593588 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 21:54:29.593786 9428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:29.583302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.125:33661 (local address 127.0.0.1:37022)
0928 21:54:29.583539 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:29.583587 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:29.583663 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:29.583807 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:29.583809 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:29.584103 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:29.584170 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:29.584172 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:29.585373 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:29.585374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:29.586215 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:29.586221 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:29.586732 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:29.586733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:29.586824 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:29.587709 (+ 885us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:29.587714 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:29.589380 (+ 1666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:29.591741 (+ 2361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:29.591756 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:29.591759 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:29.592310 (+ 551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:29.592685 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:29.592687 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:29.592688 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:29.592785 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:29.593274 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:29.593288 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:29.593368 (+ 80us) client_negotiation.cc:769] Sending connection context
0928 21:54:29.593463 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 21:54:29.593603 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":158,"threads_started":1}
I20260928 21:54:29.595434 8870 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.2.182.125:33661: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "59256c4c13d94676ac3e3b7e1e51a6a8"
member_type: NON_VOTER
last_known_addr {
host: "127.2.182.125"
port: 33661
}
attrs {
promote: true
}
}
dest_uuid: "d22a18a4a71545768146dab94c9a04a1"
cas_config_opid_index: -1
I20260928 21:54:29.595947 8870 consensus_queue.cc:237] T 00000000000000000000000000000000 P d22a18a4a71545768146dab94c9a04a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d22a18a4a71545768146dab94c9a04a1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43551 } } peers { permanent_uuid: "59256c4c13d94676ac3e3b7e1e51a6a8" member_type: NON_VOTER last_known_addr { host: "127.2.182.125" port: 33661 } attrs { promote: true } }
I20260928 21:54:29.596817 9429 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d22a18a4a71545768146dab94c9a04a1 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 59256c4c13d94676ac3e3b7e1e51a6a8 (127.2.182.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d22a18a4a71545768146dab94c9a04a1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43551 } } peers { permanent_uuid: "59256c4c13d94676ac3e3b7e1e51a6a8" member_type: NON_VOTER last_known_addr { host: "127.2.182.125" port: 33661 } attrs { promote: true } } }
I20260928 21:54:29.597436 9376 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59256c4c13d94676ac3e3b7e1e51a6a8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:54:29.598246 9429 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.2.182.125:33661
I20260928 21:54:29.598418 9429 sys_catalog.cc:455] T 00000000000000000000000000000000 P d22a18a4a71545768146dab94c9a04a1 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d22a18a4a71545768146dab94c9a04a1" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d22a18a4a71545768146dab94c9a04a1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43551 } } peers { permanent_uuid: "59256c4c13d94676ac3e3b7e1e51a6a8" member_type: NON_VOTER last_known_addr { host: "127.2.182.125" port: 33661 } attrs { promote: true } } }
I20260928 21:54:29.598577 9429 sys_catalog.cc:458] T 00000000000000000000000000000000 P d22a18a4a71545768146dab94c9a04a1 [sys.catalog]: This master's current role is: LEADER
I20260928 21:54:29.598594 9334 master.cc:537] Master@127.2.182.125:33661 shutting down...
I20260928 21:54:29.598951 9376 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 59256c4c13d94676ac3e3b7e1e51a6a8 [term 1 FOLLOWER]: Refusing update from remote peer d22a18a4a71545768146dab94c9a04a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260928 21:54:29.599460 9430 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d22a18a4a71545768146dab94c9a04a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59256c4c13d94676ac3e3b7e1e51a6a8" member_type: NON_VOTER last_known_addr { host: "127.2.182.125" port: 33661 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 21:54:29.600404 8855 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260928 21:54:29.600860 8855 consensus_peers.cc:487] T 00000000000000000000000000000000 P d22a18a4a71545768146dab94c9a04a1 -> Peer 59256c4c13d94676ac3e3b7e1e51a6a8 (127.2.182.125:33661): Couldn't send request to peer 59256c4c13d94676ac3e3b7e1e51a6a8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 21:54:29.602121 9334 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 59256c4c13d94676ac3e3b7e1e51a6a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:54:29.602239 9334 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 59256c4c13d94676ac3e3b7e1e51a6a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:54:29.602286 9334 tablet_replica.cc:330] T 00000000000000000000000000000000 P 59256c4c13d94676ac3e3b7e1e51a6a8: stopping tablet replica
I20260928 21:54:29.605710 9334 master.cc:552] Master@127.2.182.125:33661 shutdown complete.
I20260928 21:54:29.605762 9334 master_runner.cc:305] Clearing existing system tablet
I20260928 21:54:29.605892 9334 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:29.605935 9334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:29.606843 9334 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:54:29.607450 9434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:29.607570 9334 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:29.607622 9334 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 21:54:29.607641 9334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/master-1/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/master-1/wal
uuid: "59256c4c13d94676ac3e3b7e1e51a6a8"
format_stamp: "Formatted at 2026-09-28 21:54:29 on dist-test-slave-k6lz"
server_key: "5e14299801f4267c55111fb489a08437"
server_key_iv: "e88034f8c1790aa359e4c01b23857e23"
server_key_version: "encryptionkey@0"
I20260928 21:54:29.608510 9334 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 59256c4c13d94676ac3e3b7e1e51a6a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:54:29.610836 9334 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 59256c4c13d94676ac3e3b7e1e51a6a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 21:54:29.610951 9334 log.cc:1196] T 00000000000000000000000000000000 P 59256c4c13d94676ac3e3b7e1e51a6a8: Deleting WAL directory at /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260928 21:54:29.611212 9334 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 59256c4c13d94676ac3e3b7e1e51a6a8: Deleting consensus metadata
I20260928 21:54:29.611320 9334 master_runner.cc:315] Copying system tablet from 127.2.182.126:43551
I20260928 21:54:29.612196 9334 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 59256c4c13d94676ac3e3b7e1e51a6a8: tablet copy: Beginning tablet copy session from remote peer at address 127.2.182.126:43551
I20260928 21:54:29.618551 8910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:29.612751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46366 (local address 127.2.182.126:43551)
0928 21:54:29.612816 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:29.612818 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:29.612891 (+ 73us) server_negotiation.cc:407] Connection header received
0928 21:54:29.612981 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:29.612982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:29.612992 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:29.613040 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:29.613402 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:29.613941 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:29.614673 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:29.614900 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:29.614956 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:29.615902 (+ 946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:29.615904 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:29.615904 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:29.615911 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:29.617251 (+ 1340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:29.617617 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:29.617619 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:29.617619 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:29.617671 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:29.617839 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:29.617840 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:29.617840 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:29.617988 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:29.618046 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:29.618251 (+ 205us) server_negotiation.cc:299] Negotiation successful
0928 21:54:29.618453 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 21:54:29.618557 9440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:29.612558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:43551 (local address 127.0.0.1:46366)
0928 21:54:29.612826 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:29.612836 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:29.612910 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:29.613045 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:29.613046 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:29.613245 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:29.613322 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:29.613324 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:29.614031 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:29.614032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:29.614593 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:29.614595 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:29.614968 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:29.614969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:29.615034 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:29.615504 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:29.615509 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:29.615816 (+ 307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:29.617373 (+ 1557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:29.617374 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:29.617374 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:29.617548 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:29.617728 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:29.617729 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:29.617729 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:29.617786 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:29.618063 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:29.618064 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:29.618141 (+ 77us) client_negotiation.cc:769] Sending connection context
0928 21:54:29.618202 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 21:54:29.618428 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":71,"threads_started":1}
I20260928 21:54:29.618907 8890 tablet_copy_service.cc:140] P d22a18a4a71545768146dab94c9a04a1: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 59256c4c13d94676ac3e3b7e1e51a6a8 ({username='oryx', principal='oryx/127.2.182.125@KRBTEST.COM'} at 127.0.0.1:46366)
I20260928 21:54:29.619000 8890 tablet_copy_service.cc:161] P d22a18a4a71545768146dab94c9a04a1: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 59256c4c13d94676ac3e3b7e1e51a6a8 at {username='oryx', principal='oryx/127.2.182.125@KRBTEST.COM'} at 127.0.0.1:46366: session id = 59256c4c13d94676ac3e3b7e1e51a6a8-00000000000000000000000000000000
I20260928 21:54:29.619828 8890 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P d22a18a4a71545768146dab94c9a04a1: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 21:54:29.620462 9334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 21:54:29.621417 9334 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 59256c4c13d94676ac3e3b7e1e51a6a8: tablet copy: Starting download of 0 data blocks...
I20260928 21:54:29.621555 9334 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 59256c4c13d94676ac3e3b7e1e51a6a8: tablet copy: Starting download of 1 WAL segments...
I20260928 21:54:29.622839 9334 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 59256c4c13d94676ac3e3b7e1e51a6a8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 21:54:29.624233 8890 tablet_copy_service.cc:342] P d22a18a4a71545768146dab94c9a04a1: Request end of tablet copy session 59256c4c13d94676ac3e3b7e1e51a6a8-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.2.182.125@KRBTEST.COM'} at 127.0.0.1:46366
I20260928 21:54:29.624311 8890 tablet_copy_service.cc:433] P d22a18a4a71545768146dab94c9a04a1: ending tablet copy session 59256c4c13d94676ac3e3b7e1e51a6a8-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 59256c4c13d94676ac3e3b7e1e51a6a8
I20260928 21:54:29.627323 9334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:54:29.629592 9442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:29.629599 9441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:29.629657 9444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:29.630167 9334 server_base.cc:1034] running on GCE node
I20260928 21:54:29.630273 9334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:29.630465 9334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:29.631636 9334 hybrid_clock.cc:648] HybridClock initialized: now 1790632469631626 us; error 25 us; skew 500 ppm
Sep 28 21:54:29 dist-test-slave-k6lz krb5kdc[8825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632469, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:54:29.633881 9334 init.cc:377] Logged in from keytab as oryx/127.2.182.125@KRBTEST.COM (short username oryx)
I20260928 21:54:29.634225 9334 webserver.cc:460] Webserver started at http://127.2.182.125:34545/ using document root <none> and password file <none>
I20260928 21:54:29.634380 9334 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:29.634411 9334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:29.635046 9334 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:54:29.635581 9451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:29.635771 9334 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:29.635819 9334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/master-1/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/master-1/wal
uuid: "59256c4c13d94676ac3e3b7e1e51a6a8"
format_stamp: "Formatted at 2026-09-28 21:54:29 on dist-test-slave-k6lz"
server_key: "5e14299801f4267c55111fb489a08437"
server_key_iv: "e88034f8c1790aa359e4c01b23857e23"
server_key_version: "encryptionkey@0"
I20260928 21:54:29.635880 9334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:29.640518 9334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:54:29.641981 9334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:54:29.644927 9334 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.125:33661
I20260928 21:54:29.645376 9334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/master-1/data/info.pb
I20260928 21:54:29.645432 9503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.125:33661 every 8 connection(s)
I20260928 21:54:29.646246 9504 sys_catalog.cc:263] Verifying existing consensus state
I20260928 21:54:29.646741 9504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59256c4c13d94676ac3e3b7e1e51a6a8: Bootstrap starting.
I20260928 21:54:29.650713 9504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59256c4c13d94676ac3e3b7e1e51a6a8: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 21:54:29.650977 9504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59256c4c13d94676ac3e3b7e1e51a6a8: Bootstrap complete.
I20260928 21:54:29.651206 9504 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59256c4c13d94676ac3e3b7e1e51a6a8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d22a18a4a71545768146dab94c9a04a1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43551 } } peers { permanent_uuid: "59256c4c13d94676ac3e3b7e1e51a6a8" member_type: NON_VOTER last_known_addr { host: "127.2.182.125" port: 33661 } attrs { promote: true } }
I20260928 21:54:29.651302 9504 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59256c4c13d94676ac3e3b7e1e51a6a8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 59256c4c13d94676ac3e3b7e1e51a6a8, State: Initialized, Role: LEARNER
I20260928 21:54:29.651458 9504 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59256c4c13d94676ac3e3b7e1e51a6a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d22a18a4a71545768146dab94c9a04a1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43551 } } peers { permanent_uuid: "59256c4c13d94676ac3e3b7e1e51a6a8" member_type: NON_VOTER last_known_addr { host: "127.2.182.125" port: 33661 } attrs { promote: true } }
I20260928 21:54:29.651676 9507 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59256c4c13d94676ac3e3b7e1e51a6a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d22a18a4a71545768146dab94c9a04a1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43551 } } peers { permanent_uuid: "59256c4c13d94676ac3e3b7e1e51a6a8" member_type: NON_VOTER last_known_addr { host: "127.2.182.125" port: 33661 } attrs { promote: true } } }
I20260928 21:54:29.651690 9504 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59256c4c13d94676ac3e3b7e1e51a6a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:54:29.651908 9507 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59256c4c13d94676ac3e3b7e1e51a6a8 [sys.catalog]: This master's current role is: LEARNER
I20260928 21:54:29.653527 9518 catalog_manager.cc:1259] Loaded cluster ID: 088e5e211b3d41aa93ae71942e3a4aab
I20260928 21:54:29.653590 9518 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 59256c4c13d94676ac3e3b7e1e51a6a8: loading cluster ID for follower catalog manager: success
I20260928 21:54:29.654755 9518 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 59256c4c13d94676ac3e3b7e1e51a6a8: acquiring CA information for follower catalog manager: success
I20260928 21:54:29.655097 9518 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 59256c4c13d94676ac3e3b7e1e51a6a8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 21:54:29.666999 9519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:29.660449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37034 (local address 127.2.182.125:33661)
0928 21:54:29.660598 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:29.660599 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:29.660606 (+ 7us) server_negotiation.cc:407] Connection header received
0928 21:54:29.660627 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:29.660628 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:29.660636 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:29.660698 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:29.661298 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:29.661962 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:29.662711 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:29.662918 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:29.662977 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:29.664244 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:29.664246 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:29.664246 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:29.664253 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:29.665678 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:29.666074 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:29.666076 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:29.666076 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:29.666130 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:29.666310 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:29.666311 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:29.666311 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:29.666424 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:29.666515 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:29.666619 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 21:54:29.666859 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":77,"threads_started":1}
WARNING: no policy specified for oryx/127.2.182.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.182.124@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.182.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/krb5kdc/oryx_127.2.182.124.keytab.
Entry for principal oryx/127.2.182.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/krb5kdc/oryx_127.2.182.124.keytab.
I20260928 21:54:29.707450 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.124:40387
--webserver_interface=127.2.182.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/krb5kdc/oryx_127.2.182.124.keytab
--principal=oryx/127.2.182.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:45923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.126:43551,127.2.182.125:33661,127.2.182.124:40387
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:54:29.801613 9522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:54:29.801833 9522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:54:29.801856 9522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:54:29.803085 9522 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:54:29.803129 9522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:54:29.803143 9522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:54:29.803154 9522 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:54:29.803165 9522 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:54:29.805022 9522 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:45923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.126:43551,127.2.182.125:33661,127.2.182.124:40387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/krb5kdc/oryx_127.2.182.124.keytab
--principal=oryx/127.2.182.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.124:40387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.182.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9522
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:54:29.805262 9522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:54:29.805450 9522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:54:29.811519 9528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:29.811511 9530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:29.811730 9527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:29.811767 9522 server_base.cc:1034] running on GCE node
I20260928 21:54:29.812130 9522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:29.812430 9522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:29.813643 9522 hybrid_clock.cc:648] HybridClock initialized: now 1790632469813596 us; error 81 us; skew 500 ppm
Sep 28 21:54:29 dist-test-slave-k6lz krb5kdc[8825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632469, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:54:29.816998 9522 init.cc:377] Logged in from keytab as oryx/127.2.182.124@KRBTEST.COM (short username oryx)
I20260928 21:54:29.817456 9522 webserver.cc:460] Webserver started at http://127.2.182.124:33439/ using document root <none> and password file <none>
I20260928 21:54:29.817677 9522 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:29.817799 9522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:29.817934 9522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:54:29.819058 9522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/master-2/data/instance:
uuid: "8e56bb6247ad4b39a8bf4340451035a5"
format_stamp: "Formatted at 2026-09-28 21:54:29 on dist-test-slave-k6lz"
server_key: "b74083c8511c4179b92eb694a9ac4d79"
server_key_iv: "d81d518f553b8b88e9257af5fcace9bd"
server_key_version: "encryptionkey@0"
I20260928 21:54:29.819422 9522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/master-2/wal/instance:
uuid: "8e56bb6247ad4b39a8bf4340451035a5"
format_stamp: "Formatted at 2026-09-28 21:54:29 on dist-test-slave-k6lz"
server_key: "b74083c8511c4179b92eb694a9ac4d79"
server_key_iv: "d81d518f553b8b88e9257af5fcace9bd"
server_key_version: "encryptionkey@0"
I20260928 21:54:29.820775 9522 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:54:29.821517 9537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:29.821681 9522 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:54:29.821817 9522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/master-2/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/master-2/wal
uuid: "8e56bb6247ad4b39a8bf4340451035a5"
format_stamp: "Formatted at 2026-09-28 21:54:29 on dist-test-slave-k6lz"
server_key: "b74083c8511c4179b92eb694a9ac4d79"
server_key_iv: "d81d518f553b8b88e9257af5fcace9bd"
server_key_version: "encryptionkey@0"
I20260928 21:54:29.821887 9522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:29.837704 9522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:54:29.840360 9522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:54:29.840500 9522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:54:29.844645 9522 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.124:40387
I20260928 21:54:29.844686 9589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.124:40387 every 8 connection(s)
I20260928 21:54:29.845038 9522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/master-2/data/info.pb
I20260928 21:54:29.845942 9590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:54:29.847549 9590 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43551 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:54:29.853191 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 9522
Sep 28 21:54:29 dist-test-slave-k6lz krb5kdc[8825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632469, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.124@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM
I20260928 21:54:29.857988 8910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:29.848398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46370 (local address 127.2.182.126:43551)
0928 21:54:29.848522 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:29.848525 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:29.848759 (+ 234us) server_negotiation.cc:407] Connection header received
0928 21:54:29.849500 (+ 741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:29.849502 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:29.849518 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:29.849572 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:29.850328 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:29.850865 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:29.851959 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:29.852154 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:29.852233 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:29.855071 (+ 2838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:29.855073 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:29.855073 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:29.855082 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:29.856409 (+ 1327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:29.856795 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:29.856796 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:29.856797 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:29.856959 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:29.857270 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:29.857271 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:29.857271 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:29.857413 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:29.857479 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:29.857841 (+ 362us) server_negotiation.cc:299] Negotiation successful
0928 21:54:29.857917 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 21:54:29.858291 9592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:29.848236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:43551 (local address 127.0.0.1:46370)
0928 21:54:29.848521 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:29.848636 (+ 115us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:29.849335 (+ 699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:29.849661 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:29.849664 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:29.850028 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:29.850230 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:29.850234 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:29.850959 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:29.850960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:29.851805 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:29.851808 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:29.852347 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:29.852350 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:29.852642 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:29.853436 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:29.853445 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:29.854979 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:29.856488 (+ 1509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:29.856495 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:29.856497 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:29.856732 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:29.857038 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:29.857040 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:29.857040 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:29.857191 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:29.857496 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:29.857503 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:29.857675 (+ 172us) client_negotiation.cc:769] Sending connection context
0928 21:54:29.857797 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 21:54:29.857973 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":94,"threads_started":1}
I20260928 21:54:29.858952 9590 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.182.125" port: 33661 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 21:54:29 dist-test-slave-k6lz krb5kdc[8825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.182.124@KRBTEST.COM
I20260928 21:54:29.862756 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:29.853759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55066 (local address 127.2.182.124:40387)
0928 21:54:29.853984 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:29.853989 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:29.854005 (+ 16us) server_negotiation.cc:407] Connection header received
0928 21:54:29.854025 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:29.854027 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:29.854044 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:29.854116 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:29.854719 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:29.855614 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:29.856497 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:29.856854 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:29.856924 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:29.859726 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:29.859764 (+ 38us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:29.859765 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:29.859771 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:29.861328 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:29.861675 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:29.861678 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:29.861678 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:29.861744 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:29.861950 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:29.861951 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:29.861951 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:29.862138 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:29.862259 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:29.862569 (+ 310us) server_negotiation.cc:299] Negotiation successful
0928 21:54:29.862645 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":98,"threads_started":1}
Sep 28 21:54:29 dist-test-slave-k6lz krb5kdc[8825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632469, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.124@KRBTEST.COM for oryx/127.2.182.125@KRBTEST.COM
I20260928 21:54:29.866484 9592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:29.859212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.125:33661 (local address 127.0.0.1:37042)
0928 21:54:29.859258 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:29.859269 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:29.859308 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:29.859580 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:29.859582 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:29.859710 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:29.859776 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:29.859778 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:29.860664 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:29.860665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:29.861249 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:29.861251 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:29.861677 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:29.861678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:29.861739 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:29.862126 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:29.862130 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:29.863838 (+ 1708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:29.865542 (+ 1704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:29.865544 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:29.865544 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:29.865718 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:29.865902 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:29.865903 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:29.865903 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:29.865957 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:29.866171 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:29.866172 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:29.866216 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 21:54:29.866276 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 21:54:29.866379 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 21:54:29.866484 9519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:29.859372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37042 (local address 127.2.182.125:33661)
0928 21:54:29.859434 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:29.859436 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:29.859445 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:54:29.859472 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:29.859473 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:29.859487 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:29.859553 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:29.859853 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:29.860550 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:29.861346 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:29.861602 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:29.861661 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:29.863970 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:29.863973 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:29.863973 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:29.863984 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:29.865447 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:29.865794 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:29.865795 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:29.865795 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:29.865845 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:29.866006 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:29.866007 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:29.866007 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:29.866115 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:29.866169 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:29.866306 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 21:54:29.866394 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 21:54:29.868198 9590 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.182.124" port: 40387 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 21:54:29 dist-test-slave-k6lz krb5kdc[8825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632469, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.124@KRBTEST.COM for oryx/127.2.182.124@KRBTEST.COM
I20260928 21:54:29.875512 9592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:29.868495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.124:40387 (local address 127.0.0.1:55070)
0928 21:54:29.868614 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:29.868624 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:29.868660 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:29.868797 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:29.868798 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:29.868923 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:29.868961 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:29.868962 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:29.869970 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:29.869971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:29.870490 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:29.870492 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:29.870974 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:29.870975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:29.871021 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:29.871495 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:29.871500 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:29.872780 (+ 1280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:29.874298 (+ 1518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:29.874299 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:29.874299 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:29.874453 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:29.874678 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:29.874679 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:29.874679 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:29.874751 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:29.875153 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:29.875154 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:29.875200 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 21:54:29.875260 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 21:54:29.875399 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105}
I20260928 21:54:29.875588 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:29.868573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55070 (local address 127.2.182.124:40387)
0928 21:54:29.868665 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:29.868667 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:29.868676 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:54:29.868700 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:29.868701 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:29.868710 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:29.868759 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:29.869085 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:29.869873 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:29.870578 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:29.870889 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:29.870947 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:29.872908 (+ 1961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:29.872909 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:29.872910 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:29.872916 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:29.874197 (+ 1281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:29.874524 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:29.874525 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:29.874525 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:29.874576 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:29.874921 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:29.874922 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:29.874922 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:29.875067 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:29.875136 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:29.875386 (+ 250us) server_negotiation.cc:299] Negotiation successful
0928 21:54:29.875452 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 21:54:29.877012 9590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e56bb6247ad4b39a8bf4340451035a5: Bootstrap starting.
I20260928 21:54:29.877843 9590 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e56bb6247ad4b39a8bf4340451035a5: Neither blocks nor log segments found. Creating new log.
I20260928 21:54:29.878095 9590 log.cc:824] T 00000000000000000000000000000000 P 8e56bb6247ad4b39a8bf4340451035a5: Log is configured to *not* fsync() on all Append() calls
I20260928 21:54:29.878876 9590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e56bb6247ad4b39a8bf4340451035a5: No bootstrap required, opened a new log
I20260928 21:54:29.880627 9590 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e56bb6247ad4b39a8bf4340451035a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d22a18a4a71545768146dab94c9a04a1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43551 } } peers { permanent_uuid: "59256c4c13d94676ac3e3b7e1e51a6a8" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 33661 } } peers { permanent_uuid: "8e56bb6247ad4b39a8bf4340451035a5" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 40387 } }
I20260928 21:54:29.880818 9590 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e56bb6247ad4b39a8bf4340451035a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:54:29.880854 9590 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e56bb6247ad4b39a8bf4340451035a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e56bb6247ad4b39a8bf4340451035a5, State: Initialized, Role: FOLLOWER
I20260928 21:54:29.881043 9590 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e56bb6247ad4b39a8bf4340451035a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d22a18a4a71545768146dab94c9a04a1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43551 } } peers { permanent_uuid: "59256c4c13d94676ac3e3b7e1e51a6a8" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 33661 } } peers { permanent_uuid: "8e56bb6247ad4b39a8bf4340451035a5" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 40387 } }
I20260928 21:54:29.881649 9590 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e56bb6247ad4b39a8bf4340451035a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:54:29.882459 9595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e56bb6247ad4b39a8bf4340451035a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d22a18a4a71545768146dab94c9a04a1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43551 } } peers { permanent_uuid: "59256c4c13d94676ac3e3b7e1e51a6a8" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 33661 } } peers { permanent_uuid: "8e56bb6247ad4b39a8bf4340451035a5" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 40387 } } }
I20260928 21:54:29.882573 9595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e56bb6247ad4b39a8bf4340451035a5 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 21:54:29.883566 9606 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8e56bb6247ad4b39a8bf4340451035a5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:54:29.883623 9606 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:54:29.886812 9522 master_runner.cc:428] Detected that this master 8e56bb6247ad4b39a8bf4340451035a5 is joining an existing cluster
I20260928 21:54:29.886888 9522 master_runner.cc:432] Initiating AddMaster RPC to add 127.2.182.124:40387
I20260928 21:54:29.896425 9519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:29.888883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37046 (local address 127.2.182.125:33661)
0928 21:54:29.888934 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:29.888937 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:29.889105 (+ 168us) server_negotiation.cc:407] Connection header received
0928 21:54:29.889199 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:29.889200 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:29.889212 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:29.889243 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:29.889716 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:29.890507 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:29.891567 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:29.891876 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:29.891915 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:29.893395 (+ 1480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:29.893396 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:29.893397 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:29.893404 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:29.895250 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:29.895657 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:29.895658 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:29.895659 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:29.895717 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:29.895923 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:29.895924 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:29.895924 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:29.896042 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:29.896097 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:29.896236 (+ 139us) server_negotiation.cc:299] Negotiation successful
0928 21:54:29.896322 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 21:54:29.896440 9612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:29.888651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.125:33661 (local address 127.0.0.1:37046)
0928 21:54:29.889025 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:29.889037 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:29.889112 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:29.889313 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:29.889314 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:29.889558 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:29.889636 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:29.889638 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:29.890592 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:29.890593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:29.891477 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:29.891480 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:29.892000 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:29.892001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:29.892058 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:29.892959 (+ 901us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:29.892963 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:29.893302 (+ 339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:29.895395 (+ 2093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:29.895396 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:29.895396 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:29.895589 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:29.895771 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:29.895772 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:29.895772 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:29.895859 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:29.896103 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:29.896104 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:29.896148 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 21:54:29.896205 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 21:54:29.896328 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":355,"threads_started":1}
I20260928 21:54:29.914597 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 8932
I20260928 21:54:29.915544 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:29.889184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55076 (local address 127.2.182.124:40387)
0928 21:54:29.889230 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:29.889231 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:29.889420 (+ 189us) server_negotiation.cc:407] Connection header received
0928 21:54:29.889486 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:29.889486 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:29.889493 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:29.889515 (+ 22us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:29.889896 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:29.890576 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:29.891984 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:29.892222 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:29.892266 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:29.912614 (+ 20348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:29.912616 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:29.912617 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:29.912624 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:29.914313 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:29.914744 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:29.914745 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:29.914745 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:29.914804 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:29.914969 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:29.914970 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:29.914970 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:29.915134 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:29.915178 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:29.915376 (+ 198us) server_negotiation.cc:299] Negotiation successful
0928 21:54:29.915455 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 21:54:29.915704 8910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:29.888728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46386 (local address 127.2.182.126:43551)
0928 21:54:29.888781 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:29.888784 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:29.889301 (+ 517us) server_negotiation.cc:407] Connection header received
0928 21:54:29.889365 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:29.889366 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:29.889378 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:29.889445 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:29.889939 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:29.890694 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:29.891769 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:29.892058 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:29.892099 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:29.912710 (+ 20611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:29.912712 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:29.912712 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:29.912722 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:29.914438 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:29.914820 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:29.914821 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:29.914821 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:29.914871 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:29.915042 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:29.915043 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:29.915043 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:29.915155 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:29.915191 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:29.915427 (+ 236us) server_negotiation.cc:299] Negotiation successful
0928 21:54:29.915498 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 21:54:29.915905 9613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:29.889129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.124:40387 (local address 127.0.0.1:55076)
0928 21:54:29.889378 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:29.889385 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:29.889435 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:29.889564 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:29.889565 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:29.889771 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:29.889828 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:29.889830 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:29.890645 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:29.890645 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:29.891909 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:29.891911 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:29.892345 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:29.892346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:29.892408 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:29.912008 (+ 19600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:29.912016 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:29.912501 (+ 485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:29.914420 (+ 1919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:29.914421 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:29.914421 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:29.914671 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:29.914863 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:29.914864 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:29.914864 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:29.914910 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:29.915216 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:29.915218 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:29.915254 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 21:54:29.915618 (+ 364us) client_negotiation.cc:240] Negotiation successful
0928 21:54:29.915706 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":58,"threads_started":1}
I20260928 21:54:29.916095 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:29.888654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:43551 (local address 127.0.0.1:46386)
0928 21:54:29.889247 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:29.889256 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:29.889304 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:29.889489 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:29.889490 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:29.889743 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:29.889846 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:29.889849 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:29.890791 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:29.890792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:29.891673 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:29.891676 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:29.892172 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:29.892173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:29.892244 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:29.912057 (+ 19813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:29.912063 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:29.912545 (+ 482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:29.914516 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:29.914516 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:29.914517 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:29.914744 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:29.914934 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:29.914934 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:29.914934 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:29.914979 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:29.915231 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:29.915231 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:29.915291 (+ 60us) client_negotiation.cc:769] Sending connection context
0928 21:54:29.915830 (+ 539us) client_negotiation.cc:240] Negotiation successful
0928 21:54:29.915925 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"spinlock_wait_cycles":9088,"thread_start_us":306,"threads_started":1}
I20260928 21:54:29.920447 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.65:39731
--local_ip_for_outbound_sockets=127.2.182.65
--tserver_master_addrs=127.2.182.126:43551
--webserver_port=35807
--webserver_interface=127.2.182.65
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/krb5kdc/oryx_127.2.182.65.keytab
--principal=oryx/127.2.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:45923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.182.126:43551,127.2.182.125:33661
--tserver_master_addrs=127.2.182.126:43551,127.2.182.125:33661,127.2.182.124:40387 with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 21:54:29 dist-test-slave-k6lz krb5kdc[8825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632468, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.126@KRBTEST.COM for oryx/127.2.182.124@KRBTEST.COM
I20260928 21:54:29.929780 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:29.921774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55088 (local address 127.2.182.124:40387)
0928 21:54:29.921807 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:29.921809 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:29.921830 (+ 21us) server_negotiation.cc:407] Connection header received
0928 21:54:29.921901 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:29.921902 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:29.921911 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:29.921956 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:29.922342 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:29.923078 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:29.924293 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:29.924513 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:29.924696 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:29.926906 (+ 2210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:29.926907 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:29.926908 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:29.926915 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:29.928481 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:29.928823 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:29.928824 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:29.928825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:29.928960 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:29.929144 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:29.929145 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:29.929145 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:29.929260 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:29.929313 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:29.929601 (+ 288us) server_negotiation.cc:299] Negotiation successful
0928 21:54:29.929674 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 21:54:29.929780 9428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:29.921732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.124:40387 (local address 127.0.0.1:55088)
0928 21:54:29.921777 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:29.921788 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:29.921846 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:29.922008 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:29.922009 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:29.922197 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:29.922252 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:29.922254 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:29.923257 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:29.923258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:29.924157 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:29.924166 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:29.924691 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:29.924692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:29.924907 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:29.925387 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:29.925392 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:29.926805 (+ 1413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:29.928577 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:29.928578 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:29.928579 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:29.928753 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:29.929027 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:29.929028 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:29.929028 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:29.929089 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:29.929316 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:29.929317 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:29.929456 (+ 139us) client_negotiation.cc:769] Sending connection context
0928 21:54:29.929545 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 21:54:29.929675 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 21:54:29.930624 8870 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.2.182.124:40387: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "8e56bb6247ad4b39a8bf4340451035a5"
member_type: NON_VOTER
last_known_addr {
host: "127.2.182.124"
port: 40387
}
attrs {
promote: true
}
}
dest_uuid: "d22a18a4a71545768146dab94c9a04a1"
cas_config_opid_index: 5
I20260928 21:54:29.930891 8870 consensus_queue.cc:237] T 00000000000000000000000000000000 P d22a18a4a71545768146dab94c9a04a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d22a18a4a71545768146dab94c9a04a1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43551 } } peers { permanent_uuid: "59256c4c13d94676ac3e3b7e1e51a6a8" member_type: NON_VOTER last_known_addr { host: "127.2.182.125" port: 33661 } attrs { promote: true } } peers { permanent_uuid: "8e56bb6247ad4b39a8bf4340451035a5" member_type: NON_VOTER last_known_addr { host: "127.2.182.124" port: 40387 } attrs { promote: true } }
I20260928 21:54:29.931366 9430 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d22a18a4a71545768146dab94c9a04a1 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 8e56bb6247ad4b39a8bf4340451035a5 (127.2.182.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d22a18a4a71545768146dab94c9a04a1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43551 } } peers { permanent_uuid: "59256c4c13d94676ac3e3b7e1e51a6a8" member_type: NON_VOTER last_known_addr { host: "127.2.182.125" port: 33661 } attrs { promote: true } } peers { permanent_uuid: "8e56bb6247ad4b39a8bf4340451035a5" member_type: NON_VOTER last_known_addr { host: "127.2.182.124" port: 40387 } attrs { promote: true } } }
I20260928 21:54:29.932199 9430 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.2.182.124:40387
I20260928 21:54:29.932274 9430 sys_catalog.cc:455] T 00000000000000000000000000000000 P d22a18a4a71545768146dab94c9a04a1 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d22a18a4a71545768146dab94c9a04a1" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d22a18a4a71545768146dab94c9a04a1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43551 } } peers { permanent_uuid: "59256c4c13d94676ac3e3b7e1e51a6a8" member_type: NON_VOTER last_known_addr { host: "127.2.182.125" port: 33661 } attrs { promote: true } } peers { permanent_uuid: "8e56bb6247ad4b39a8bf4340451035a5" member_type: NON_VOTER last_known_addr { host: "127.2.182.124" port: 40387 } attrs { promote: true } } }
I20260928 21:54:29.932345 9430 sys_catalog.cc:458] T 00000000000000000000000000000000 P d22a18a4a71545768146dab94c9a04a1 [sys.catalog]: This master's current role is: LEADER
I20260928 21:54:29.932680 9522 master.cc:537] Master@127.2.182.124:40387 shutting down...
W20260928 21:54:29.933528 8855 consensus_peers.cc:487] T 00000000000000000000000000000000 P d22a18a4a71545768146dab94c9a04a1 -> Peer 8e56bb6247ad4b39a8bf4340451035a5 (127.2.182.124:40387): Couldn't send request to peer 8e56bb6247ad4b39a8bf4340451035a5. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 21:54:29.937016 9519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:29.933145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37056 (local address 127.2.182.125:33661)
0928 21:54:29.933209 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:29.933211 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:29.933218 (+ 7us) server_negotiation.cc:407] Connection header received
0928 21:54:29.933241 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:29.933242 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:29.933267 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:29.933330 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:54:29.933804 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:29.934508 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:29.935880 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:29.936408 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:29.936489 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:29.936559 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:29.936910 (+ 351us) server_negotiation.cc:299] Negotiation successful
0928 21:54:29.936949 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 21:54:29.937016 9428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:29.932741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.125:33661 (local address 127.0.0.1:37056)
0928 21:54:29.932966 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:29.932976 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:29.933009 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:29.933682 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:29.933682 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:29.933684 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:54:29.933727 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:29.933728 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:29.934591 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:29.934592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:29.935780 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:29.935783 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:29.936754 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:29.936755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:29.936823 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:29.936839 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 21:54:29.936890 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 21:54:29.936937 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206}
I20260928 21:54:29.937390 9478 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 59256c4c13d94676ac3e3b7e1e51a6a8 [term 1 LEARNER]: Refusing update from remote peer d22a18a4a71545768146dab94c9a04a1: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 21:54:29.937815 9522 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 8e56bb6247ad4b39a8bf4340451035a5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260928 21:54:29.937909 9522 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 8e56bb6247ad4b39a8bf4340451035a5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260928 21:54:29.937958 9522 tablet_replica.cc:330] T 00000000000000000000000000000000 P 8e56bb6247ad4b39a8bf4340451035a5: stopping tablet replica
I20260928 21:54:29.937597 9616 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d22a18a4a71545768146dab94c9a04a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59256c4c13d94676ac3e3b7e1e51a6a8" member_type: NON_VOTER last_known_addr { host: "127.2.182.125" port: 33661 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260928 21:54:29.939620 9478 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 59256c4c13d94676ac3e3b7e1e51a6a8 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 8e56bb6247ad4b39a8bf4340451035a5 (127.2.182.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d22a18a4a71545768146dab94c9a04a1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43551 } } peers { permanent_uuid: "59256c4c13d94676ac3e3b7e1e51a6a8" member_type: NON_VOTER last_known_addr { host: "127.2.182.125" port: 33661 } attrs { promote: true } } peers { permanent_uuid: "8e56bb6247ad4b39a8bf4340451035a5" member_type: NON_VOTER last_known_addr { host: "127.2.182.124" port: 40387 } attrs { promote: true } } }
I20260928 21:54:29.940760 9507 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59256c4c13d94676ac3e3b7e1e51a6a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d22a18a4a71545768146dab94c9a04a1. Latest consensus state: current_term: 1 leader_uuid: "d22a18a4a71545768146dab94c9a04a1" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d22a18a4a71545768146dab94c9a04a1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43551 } } peers { permanent_uuid: "59256c4c13d94676ac3e3b7e1e51a6a8" member_type: NON_VOTER last_known_addr { host: "127.2.182.125" port: 33661 } attrs { promote: true } } peers { permanent_uuid: "8e56bb6247ad4b39a8bf4340451035a5" member_type: NON_VOTER last_known_addr { host: "127.2.182.124" port: 40387 } attrs { promote: true } } }
I20260928 21:54:29.940876 9507 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59256c4c13d94676ac3e3b7e1e51a6a8 [sys.catalog]: This master's current role is: LEARNER
I20260928 21:54:29.940763 9618 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59256c4c13d94676ac3e3b7e1e51a6a8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d22a18a4a71545768146dab94c9a04a1" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d22a18a4a71545768146dab94c9a04a1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43551 } } peers { permanent_uuid: "59256c4c13d94676ac3e3b7e1e51a6a8" member_type: NON_VOTER last_known_addr { host: "127.2.182.125" port: 33661 } attrs { promote: true } } peers { permanent_uuid: "8e56bb6247ad4b39a8bf4340451035a5" member_type: NON_VOTER last_known_addr { host: "127.2.182.124" port: 40387 } attrs { promote: true } } }
I20260928 21:54:29.941077 9618 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59256c4c13d94676ac3e3b7e1e51a6a8 [sys.catalog]: This master's current role is: LEARNER
I20260928 21:54:29.941212 9616 raft_consensus.cc:1050] T 00000000000000000000000000000000 P d22a18a4a71545768146dab94c9a04a1: attempting to promote NON_VOTER 59256c4c13d94676ac3e3b7e1e51a6a8 to VOTER
I20260928 21:54:29.941411 9616 consensus_queue.cc:237] T 00000000000000000000000000000000 P d22a18a4a71545768146dab94c9a04a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d22a18a4a71545768146dab94c9a04a1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43551 } } peers { permanent_uuid: "59256c4c13d94676ac3e3b7e1e51a6a8" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 33661 } attrs { promote: false } } peers { permanent_uuid: "8e56bb6247ad4b39a8bf4340451035a5" member_type: NON_VOTER last_known_addr { host: "127.2.182.124" port: 40387 } attrs { promote: true } }
I20260928 21:54:29.941885 9428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:29.941738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.124:40387 (local address 127.0.0.1:55092)
0928 21:54:29.941807 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:29.941853 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.124:40387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 21:54:29.941955 9478 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 59256c4c13d94676ac3e3b7e1e51a6a8 [term 1 LEARNER]: Refusing update from remote peer d22a18a4a71545768146dab94c9a04a1: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260928 21:54:29.942130 9616 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d22a18a4a71545768146dab94c9a04a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59256c4c13d94676ac3e3b7e1e51a6a8" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 33661 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260928 21:54:29.942332 9428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:29.942249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.124:40387 (local address 127.0.0.1:55100)
0928 21:54:29.942277 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:29.942310 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.124:40387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260928 21:54:29.942443 8855 consensus_peers.cc:487] T 00000000000000000000000000000000 P d22a18a4a71545768146dab94c9a04a1 -> Peer 8e56bb6247ad4b39a8bf4340451035a5 (127.2.182.124:40387): Couldn't send request to peer 8e56bb6247ad4b39a8bf4340451035a5. Status: Network error: Client connection negotiation failed: client connection to 127.2.182.124:40387: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 21:54:29.942749 9429 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d22a18a4a71545768146dab94c9a04a1 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 59256c4c13d94676ac3e3b7e1e51a6a8 (127.2.182.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d22a18a4a71545768146dab94c9a04a1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43551 } } peers { permanent_uuid: "59256c4c13d94676ac3e3b7e1e51a6a8" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 33661 } attrs { promote: false } } peers { permanent_uuid: "8e56bb6247ad4b39a8bf4340451035a5" member_type: NON_VOTER last_known_addr { host: "127.2.182.124" port: 40387 } attrs { promote: true } } }
I20260928 21:54:29.942925 9478 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 59256c4c13d94676ac3e3b7e1e51a6a8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 59256c4c13d94676ac3e3b7e1e51a6a8 (127.2.182.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d22a18a4a71545768146dab94c9a04a1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43551 } } peers { permanent_uuid: "59256c4c13d94676ac3e3b7e1e51a6a8" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 33661 } attrs { promote: false } } peers { permanent_uuid: "8e56bb6247ad4b39a8bf4340451035a5" member_type: NON_VOTER last_known_addr { host: "127.2.182.124" port: 40387 } attrs { promote: true } } }
I20260928 21:54:29.943879 9429 sys_catalog.cc:455] T 00000000000000000000000000000000 P d22a18a4a71545768146dab94c9a04a1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d22a18a4a71545768146dab94c9a04a1" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d22a18a4a71545768146dab94c9a04a1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43551 } } peers { permanent_uuid: "59256c4c13d94676ac3e3b7e1e51a6a8" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 33661 } attrs { promote: false } } peers { permanent_uuid: "8e56bb6247ad4b39a8bf4340451035a5" member_type: NON_VOTER last_known_addr { host: "127.2.182.124" port: 40387 } attrs { promote: true } } }
I20260928 21:54:29.943960 9429 sys_catalog.cc:458] T 00000000000000000000000000000000 P d22a18a4a71545768146dab94c9a04a1 [sys.catalog]: This master's current role is: LEADER
I20260928 21:54:29.944024 9430 sys_catalog.cc:455] T 00000000000000000000000000000000 P d22a18a4a71545768146dab94c9a04a1 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d22a18a4a71545768146dab94c9a04a1" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d22a18a4a71545768146dab94c9a04a1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43551 } } peers { permanent_uuid: "59256c4c13d94676ac3e3b7e1e51a6a8" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 33661 } attrs { promote: false } } peers { permanent_uuid: "8e56bb6247ad4b39a8bf4340451035a5" member_type: NON_VOTER last_known_addr { host: "127.2.182.124" port: 40387 } attrs { promote: true } } }
I20260928 21:54:29.944010 9618 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59256c4c13d94676ac3e3b7e1e51a6a8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d22a18a4a71545768146dab94c9a04a1" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d22a18a4a71545768146dab94c9a04a1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43551 } } peers { permanent_uuid: "59256c4c13d94676ac3e3b7e1e51a6a8" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 33661 } attrs { promote: false } } peers { permanent_uuid: "8e56bb6247ad4b39a8bf4340451035a5" member_type: NON_VOTER last_known_addr { host: "127.2.182.124" port: 40387 } attrs { promote: true } } }
I20260928 21:54:29.944067 9430 sys_catalog.cc:458] T 00000000000000000000000000000000 P d22a18a4a71545768146dab94c9a04a1 [sys.catalog]: This master's current role is: LEADER
I20260928 21:54:29.944072 9618 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59256c4c13d94676ac3e3b7e1e51a6a8 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:54:29.952718 9522 master.cc:552] Master@127.2.182.124:40387 shutdown complete.
I20260928 21:54:29.952795 9522 master_runner.cc:305] Clearing existing system tablet
I20260928 21:54:29.953016 9522 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:29.953073 9522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:29.954007 9522 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:54:29.954836 9621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:29.955044 9522 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:29.955109 9522 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:54:29.955132 9522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/master-2/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/master-2/wal
uuid: "8e56bb6247ad4b39a8bf4340451035a5"
format_stamp: "Formatted at 2026-09-28 21:54:29 on dist-test-slave-k6lz"
server_key: "b74083c8511c4179b92eb694a9ac4d79"
server_key_iv: "d81d518f553b8b88e9257af5fcace9bd"
server_key_version: "encryptionkey@0"
I20260928 21:54:29.955856 9522 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 8e56bb6247ad4b39a8bf4340451035a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:54:29.957624 9522 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 8e56bb6247ad4b39a8bf4340451035a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 21:54:29.957710 9522 log.cc:1196] T 00000000000000000000000000000000 P 8e56bb6247ad4b39a8bf4340451035a5: Deleting WAL directory at /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260928 21:54:29.957922 9522 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 8e56bb6247ad4b39a8bf4340451035a5: Deleting consensus metadata
I20260928 21:54:29.958029 9522 master_runner.cc:315] Copying system tablet from 127.2.182.126:43551
I20260928 21:54:29.958693 9522 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 8e56bb6247ad4b39a8bf4340451035a5: tablet copy: Beginning tablet copy session from remote peer at address 127.2.182.126:43551
I20260928 21:54:29.965596 9627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:29.959013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:43551 (local address 127.0.0.1:46396)
0928 21:54:29.959150 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:29.959160 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:29.959222 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:29.959376 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:29.959377 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:29.959560 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:29.959600 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:29.959602 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:29.960673 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:29.960675 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:29.961236 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:29.961238 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:29.961613 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:29.961614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:29.961676 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:29.962123 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:29.962129 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:29.962440 (+ 311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:29.963959 (+ 1519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:29.963960 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:29.963960 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:29.964332 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:29.964718 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:29.964720 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:29.964720 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:29.964790 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:29.965095 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:29.965096 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:29.965157 (+ 61us) client_negotiation.cc:769] Sending connection context
0928 21:54:29.965225 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 21:54:29.965377 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":69,"threads_started":1}
I20260928 21:54:29.966074 8910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:29.959083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46396 (local address 127.2.182.126:43551)
0928 21:54:29.959134 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:29.959136 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:29.959205 (+ 69us) server_negotiation.cc:407] Connection header received
0928 21:54:29.959294 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:29.959295 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:29.959305 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:29.959361 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:29.959675 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:29.960438 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:29.961323 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:29.961534 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:29.961593 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:29.962529 (+ 936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:29.962531 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:29.962531 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:29.962538 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:29.963881 (+ 1343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:29.964518 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:29.964520 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:29.964521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:29.964641 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:29.964853 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:29.964854 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:29.964855 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:29.965027 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:29.965102 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:29.965351 (+ 249us) server_negotiation.cc:299] Negotiation successful
0928 21:54:29.965615 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 21:54:29.966262 8890 tablet_copy_service.cc:140] P d22a18a4a71545768146dab94c9a04a1: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 8e56bb6247ad4b39a8bf4340451035a5 ({username='oryx', principal='oryx/127.2.182.124@KRBTEST.COM'} at 127.0.0.1:46396)
I20260928 21:54:29.966308 8890 tablet_copy_service.cc:161] P d22a18a4a71545768146dab94c9a04a1: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 8e56bb6247ad4b39a8bf4340451035a5 at {username='oryx', principal='oryx/127.2.182.124@KRBTEST.COM'} at 127.0.0.1:46396: session id = 8e56bb6247ad4b39a8bf4340451035a5-00000000000000000000000000000000
I20260928 21:54:29.966761 8890 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P d22a18a4a71545768146dab94c9a04a1: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 21:54:29.967212 9522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 21:54:29.968294 9522 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 8e56bb6247ad4b39a8bf4340451035a5: tablet copy: Starting download of 0 data blocks...
I20260928 21:54:29.968423 9522 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 8e56bb6247ad4b39a8bf4340451035a5: tablet copy: Starting download of 1 WAL segments...
I20260928 21:54:29.969115 9522 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 8e56bb6247ad4b39a8bf4340451035a5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 21:54:29.970444 8890 tablet_copy_service.cc:342] P d22a18a4a71545768146dab94c9a04a1: Request end of tablet copy session 8e56bb6247ad4b39a8bf4340451035a5-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.2.182.124@KRBTEST.COM'} at 127.0.0.1:46396
I20260928 21:54:29.970511 8890 tablet_copy_service.cc:433] P d22a18a4a71545768146dab94c9a04a1: ending tablet copy session 8e56bb6247ad4b39a8bf4340451035a5-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 8e56bb6247ad4b39a8bf4340451035a5
I20260928 21:54:29.974756 9522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:54:29.976177 9631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:29.976382 9629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:29.976501 9628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:29.983316 9522 server_base.cc:1034] running on GCE node
I20260928 21:54:29.983498 9522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:29.983649 9522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:29.984790 9522 hybrid_clock.cc:648] HybridClock initialized: now 1790632469984771 us; error 32 us; skew 500 ppm
Sep 28 21:54:29 dist-test-slave-k6lz krb5kdc[8825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632469, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:54:29.986284 9522 init.cc:377] Logged in from keytab as oryx/127.2.182.124@KRBTEST.COM (short username oryx)
I20260928 21:54:29.986548 9522 webserver.cc:460] Webserver started at http://127.2.182.124:36473/ using document root <none> and password file <none>
I20260928 21:54:29.986661 9522 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:29.986702 9522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:29.987423 9522 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:54:29.988344 9638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:29.988476 9522 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:54:29.988514 9522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/master-2/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/master-2/wal
uuid: "8e56bb6247ad4b39a8bf4340451035a5"
format_stamp: "Formatted at 2026-09-28 21:54:29 on dist-test-slave-k6lz"
server_key: "b74083c8511c4179b92eb694a9ac4d79"
server_key_iv: "d81d518f553b8b88e9257af5fcace9bd"
server_key_version: "encryptionkey@0"
I20260928 21:54:29.988582 9522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:29.994611 9522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:54:29.996142 9522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:54:29.998915 9522 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.124:40387
I20260928 21:54:29.999248 9522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/master-2/data/info.pb
I20260928 21:54:30.000114 9691 sys_catalog.cc:263] Verifying existing consensus state
I20260928 21:54:30.000514 9691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e56bb6247ad4b39a8bf4340451035a5: Bootstrap starting.
I20260928 21:54:30.001765 9690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.124:40387 every 8 connection(s)
I20260928 21:54:30.003242 9691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e56bb6247ad4b39a8bf4340451035a5: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 21:54:30.003500 9691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e56bb6247ad4b39a8bf4340451035a5: Bootstrap complete.
I20260928 21:54:30.003702 9691 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e56bb6247ad4b39a8bf4340451035a5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d22a18a4a71545768146dab94c9a04a1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43551 } } peers { permanent_uuid: "59256c4c13d94676ac3e3b7e1e51a6a8" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 33661 } attrs { promote: false } } peers { permanent_uuid: "8e56bb6247ad4b39a8bf4340451035a5" member_type: NON_VOTER last_known_addr { host: "127.2.182.124" port: 40387 } attrs { promote: true } }
I20260928 21:54:30.003791 9691 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e56bb6247ad4b39a8bf4340451035a5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8e56bb6247ad4b39a8bf4340451035a5, State: Initialized, Role: LEARNER
I20260928 21:54:30.003834 9691 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e56bb6247ad4b39a8bf4340451035a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d22a18a4a71545768146dab94c9a04a1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43551 } } peers { permanent_uuid: "59256c4c13d94676ac3e3b7e1e51a6a8" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 33661 } attrs { promote: false } } peers { permanent_uuid: "8e56bb6247ad4b39a8bf4340451035a5" member_type: NON_VOTER last_known_addr { host: "127.2.182.124" port: 40387 } attrs { promote: true } }
I20260928 21:54:30.004035 9691 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e56bb6247ad4b39a8bf4340451035a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:54:30.004005 9694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e56bb6247ad4b39a8bf4340451035a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d22a18a4a71545768146dab94c9a04a1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43551 } } peers { permanent_uuid: "59256c4c13d94676ac3e3b7e1e51a6a8" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 33661 } attrs { promote: false } } peers { permanent_uuid: "8e56bb6247ad4b39a8bf4340451035a5" member_type: NON_VOTER last_known_addr { host: "127.2.182.124" port: 40387 } attrs { promote: true } } }
I20260928 21:54:30.004185 9694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e56bb6247ad4b39a8bf4340451035a5 [sys.catalog]: This master's current role is: LEARNER
I20260928 21:54:30.005467 9705 catalog_manager.cc:1259] Loaded cluster ID: 088e5e211b3d41aa93ae71942e3a4aab
I20260928 21:54:30.006469 9705 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8e56bb6247ad4b39a8bf4340451035a5: loading cluster ID for follower catalog manager: success
I20260928 21:54:30.007845 9705 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8e56bb6247ad4b39a8bf4340451035a5: acquiring CA information for follower catalog manager: success
I20260928 21:54:30.008152 9705 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8e56bb6247ad4b39a8bf4340451035a5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 21:54:30.054396 9615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:54:30.054724 9615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:54:30.054766 9615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:54:30.056730 9615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:54:30.056818 9615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:54:30.056862 9615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.65
I20260928 21:54:30.059248 9615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:45923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/krb5kdc/oryx_127.2.182.65.keytab
--principal=oryx/127.2.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.65:39731
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.182.65
--webserver_port=35807
--tserver_master_addrs=127.2.182.126:43551,127.2.182.125:33661,127.2.182.124:40387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.65
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:54:30.059556 9615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:54:30.059840 9615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:54:30.065606 9710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:30.065755 9711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:30.065910 9615 server_base.cc:1034] running on GCE node
W20260928 21:54:30.065604 9713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:30.066202 9615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:30.066461 9615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:30.067611 9615 hybrid_clock.cc:648] HybridClock initialized: now 1790632470067596 us; error 32 us; skew 500 ppm
Sep 28 21:54:30 dist-test-slave-k6lz krb5kdc[8825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632470, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:54:30.070024 9615 init.cc:377] Logged in from keytab as oryx/127.2.182.65@KRBTEST.COM (short username oryx)
I20260928 21:54:30.070480 9615 webserver.cc:460] Webserver started at http://127.2.182.65:35807/ using document root <none> and password file <none>
I20260928 21:54:30.070677 9615 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:30.070732 9615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:30.072167 9615 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:54:30.072898 9720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:30.073130 9615 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:54:30.073230 9615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-0/wal
uuid: "4438ab182bef4715bd7a150fab1403bd"
format_stamp: "Formatted at 2026-09-28 21:54:28 on dist-test-slave-k6lz"
server_key: "ea0a718ca73f9257b2b72d1c45d1cc86"
server_key_iv: "eefcb0cec937eda91eee238205e9eded"
server_key_version: "encryptionkey@0"
I20260928 21:54:30.073662 9615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:30.083477 9615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:54:30.085631 9615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:54:30.085772 9615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:54:30.086000 9615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:54:30.086458 9615 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:54:30.086500 9615 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:30.086532 9615 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:54:30.086549 9615 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:30.092703 9615 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.65:39731
I20260928 21:54:30.092828 9833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.65:39731 every 8 connection(s)
I20260928 21:54:30.093107 9615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
I20260928 21:54:30.097688 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 9615
I20260928 21:54:30.097847 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 9066
Sep 28 21:54:30 dist-test-slave-k6lz krb5kdc[8825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632470, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.65@KRBTEST.COM for oryx/127.2.182.124@KRBTEST.COM
Sep 28 21:54:30 dist-test-slave-k6lz krb5kdc[8825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632470, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.65@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM
Sep 28 21:54:30 dist-test-slave-k6lz krb5kdc[8825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632470, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.65@KRBTEST.COM for oryx/127.2.182.125@KRBTEST.COM
I20260928 21:54:30.109583 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.66:33051
--local_ip_for_outbound_sockets=127.2.182.66
--tserver_master_addrs=127.2.182.126:43551
--webserver_port=34861
--webserver_interface=127.2.182.66
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/krb5kdc/oryx_127.2.182.66.keytab
--principal=oryx/127.2.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:45923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.182.126:43551,127.2.182.125:33661
--tserver_master_addrs=127.2.182.126:43551,127.2.182.125:33661,127.2.182.124:40387 with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:54:30.109730 8910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:30.093913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.65:60909 (local address 127.2.182.126:43551)
0928 21:54:30.093962 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:30.093965 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:30.094264 (+ 299us) server_negotiation.cc:407] Connection header received
0928 21:54:30.095297 (+ 1033us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:30.095299 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:30.095313 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:30.095390 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:30.096319 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:30.096860 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:30.098230 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:30.098508 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:30.098547 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:30.103398 (+ 4851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:30.103400 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:30.103401 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:30.103412 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:30.107771 (+ 4359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:30.108573 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:30.108575 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:30.108575 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:30.108639 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:30.109009 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:30.109010 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:30.109010 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:30.109172 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:30.109213 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:30.109566 (+ 353us) server_negotiation.cc:299] Negotiation successful
0928 21:54:30.109649 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 21:54:30.109732 9840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:30.093908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.65:58931 (local address 127.2.182.124:40387)
0928 21:54:30.094017 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:30.094019 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:30.094369 (+ 350us) server_negotiation.cc:407] Connection header received
0928 21:54:30.095377 (+ 1008us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:30.095378 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:30.095392 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:30.095435 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:30.096339 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:30.097296 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:30.098232 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:30.098515 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:30.098552 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:30.103465 (+ 4913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:30.103467 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:30.103467 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:30.103476 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:30.107651 (+ 4175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:30.108576 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:30.108577 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:30.108577 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:30.108655 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:30.108947 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:30.108948 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:30.108949 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:30.109089 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:30.109130 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:30.109539 (+ 409us) server_negotiation.cc:299] Negotiation successful
0928 21:54:30.109628 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":64,"threads_started":1}
I20260928 21:54:30.109731 9519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:30.094223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.65:34027 (local address 127.2.182.125:33661)
0928 21:54:30.094267 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:30.094269 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:30.094315 (+ 46us) server_negotiation.cc:407] Connection header received
0928 21:54:30.095379 (+ 1064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:30.095381 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:30.095392 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:30.095436 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:30.096316 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:30.097098 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:30.100369 (+ 3271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:30.100843 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:30.100952 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:30.103123 (+ 2171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:30.103125 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:30.103125 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:30.103134 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:30.107131 (+ 3997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:30.108575 (+ 1444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:30.108577 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:30.108577 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:30.108655 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:30.108948 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:30.108949 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:30.108949 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:30.109097 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:30.109134 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:30.109539 (+ 405us) server_negotiation.cc:299] Negotiation successful
0928 21:54:30.109626 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 21:54:30.110319 9839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:30.094154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.125:33661 (local address 127.2.182.65:34027)
0928 21:54:30.094267 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:30.094275 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:30.095177 (+ 902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:30.095523 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:30.095528 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:30.095966 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:30.096212 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:30.096217 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:30.097185 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:30.097186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:30.100248 (+ 3062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:30.100251 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:30.101020 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:30.101021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:30.101085 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:30.101515 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:30.101520 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:30.103063 (+ 1543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:30.107353 (+ 4290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:30.107367 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:30.107370 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:30.108475 (+ 1105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:30.108727 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:30.108728 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:30.108728 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:30.108873 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:30.109175 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:30.109182 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:30.109380 (+ 198us) client_negotiation.cc:769] Sending connection context
0928 21:54:30.109841 (+ 461us) client_negotiation.cc:240] Negotiation successful
0928 21:54:30.110131 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"mutex_wait_us":184,"thread_start_us":56,"threads_started":1}
I20260928 21:54:30.110322 9838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:30.093864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:43551 (local address 127.2.182.65:60909)
0928 21:54:30.094129 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:30.094172 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:30.095197 (+ 1025us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:30.095524 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:30.095528 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:30.095976 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:30.096213 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:30.096219 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:30.096961 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:30.096962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:30.098134 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:30.098136 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:30.098635 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:30.098636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:30.098781 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:30.100402 (+ 1621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:30.100407 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:30.102925 (+ 2518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:30.107881 (+ 4956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:30.107882 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:30.107882 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:30.108476 (+ 594us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:30.108703 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:30.108704 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:30.108704 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:30.108907 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:30.109256 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:30.109256 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:30.109436 (+ 180us) client_negotiation.cc:769] Sending connection context
0928 21:54:30.109840 (+ 404us) client_negotiation.cc:240] Negotiation successful
0928 21:54:30.110215 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":220,"thread_start_us":88,"threads_started":1}
I20260928 21:54:30.110319 9841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:30.093864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.124:40387 (local address 127.2.182.65:58931)
0928 21:54:30.094251 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:30.094261 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:30.095192 (+ 931us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:30.095523 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:30.095528 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:30.095966 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:30.096213 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:30.096219 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:30.097371 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:30.097372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:30.098137 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:30.098139 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:30.098636 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:30.098637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:30.098781 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:30.099420 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:30.099431 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:30.103358 (+ 3927us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:30.107744 (+ 4386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:30.107745 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:30.107746 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:30.108475 (+ 729us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:30.108727 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:30.108728 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:30.108728 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:30.108873 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:30.109175 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:30.109182 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:30.109380 (+ 198us) client_negotiation.cc:769] Sending connection context
0928 21:54:30.109841 (+ 461us) client_negotiation.cc:240] Negotiation successful
0928 21:54:30.110057 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"mutex_wait_us":39,"thread_start_us":87,"threads_started":1}
I20260928 21:54:30.110936 9836 heartbeater.cc:347] Connected to a master server at 127.2.182.125:33661
I20260928 21:54:30.111045 9834 heartbeater.cc:347] Connected to a master server at 127.2.182.124:40387
I20260928 21:54:30.111125 9834 heartbeater.cc:464] Registering TS with master...
I20260928 21:54:30.111128 9836 heartbeater.cc:464] Registering TS with master...
I20260928 21:54:30.111364 9834 heartbeater.cc:511] Master 127.2.182.124:40387 requested a full tablet report, sending...
I20260928 21:54:30.111352 9836 heartbeater.cc:511] Master 127.2.182.125:33661 requested a full tablet report, sending...
I20260928 21:54:30.111904 9655 ts_manager.cc:194] Registered new tserver with Master: 4438ab182bef4715bd7a150fab1403bd (127.2.182.65:39731)
I20260928 21:54:30.111907 9468 ts_manager.cc:194] Registered new tserver with Master: 4438ab182bef4715bd7a150fab1403bd (127.2.182.65:39731)
I20260928 21:54:30.113490 9835 heartbeater.cc:347] Connected to a master server at 127.2.182.126:43551
I20260928 21:54:30.113557 9835 heartbeater.cc:464] Registering TS with master...
I20260928 21:54:30.113638 9835 heartbeater.cc:511] Master 127.2.182.126:43551 requested a full tablet report, sending...
I20260928 21:54:30.113981 8870 ts_manager.cc:194] Re-registered known tserver with Master: 4438ab182bef4715bd7a150fab1403bd (127.2.182.65:39731)
I20260928 21:54:30.114355 8870 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.182.65@KRBTEST.COM'} at 127.2.182.65:60909
W20260928 21:54:30.194972 9842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:54:30.195181 9842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:54:30.195205 9842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:54:30.196446 9842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:54:30.196491 9842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:54:30.196518 9842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.66
I20260928 21:54:30.198030 9842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:45923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/krb5kdc/oryx_127.2.182.66.keytab
--principal=oryx/127.2.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.66:33051
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.182.66
--webserver_port=34861
--tserver_master_addrs=127.2.182.126:43551,127.2.182.125:33661,127.2.182.124:40387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.66
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:54:30.198271 9842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:54:30.198514 9842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:54:30.203924 9848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:30.203935 9850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:30.204087 9842 server_base.cc:1034] running on GCE node
W20260928 21:54:30.203935 9847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:30.204504 9842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:30.204761 9842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:30.205912 9842 hybrid_clock.cc:648] HybridClock initialized: now 1790632470205901 us; error 40 us; skew 500 ppm
Sep 28 21:54:30 dist-test-slave-k6lz krb5kdc[8825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632470, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:54:30.208868 9842 init.cc:377] Logged in from keytab as oryx/127.2.182.66@KRBTEST.COM (short username oryx)
I20260928 21:54:30.209360 9842 webserver.cc:460] Webserver started at http://127.2.182.66:34861/ using document root <none> and password file <none>
I20260928 21:54:30.209583 9842 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:30.209647 9842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:30.211112 9842 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:54:30.211948 9857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:30.212185 9842 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 21:54:30.212276 9842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-1/wal
uuid: "d6cf3fea18684e48bc758ab6bf07563e"
format_stamp: "Formatted at 2026-09-28 21:54:29 on dist-test-slave-k6lz"
server_key: "f0b8c51e5d3fc40804660103806ae165"
server_key_iv: "893f03ce1c74ec9a1f609459ef8cfefe"
server_key_version: "encryptionkey@0"
I20260928 21:54:30.212692 9842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:30.236737 9842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:54:30.238834 9842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:54:30.238973 9842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:54:30.239197 9842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:54:30.239535 9842 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:54:30.239570 9842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:30.239600 9842 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:54:30.239617 9842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:30.245239 9842 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.66:33051
I20260928 21:54:30.245296 9970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.66:33051 every 8 connection(s)
I20260928 21:54:30.245685 9842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
Sep 28 21:54:30 dist-test-slave-k6lz krb5kdc[8825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632470, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.66@KRBTEST.COM for oryx/127.2.182.125@KRBTEST.COM
Sep 28 21:54:30 dist-test-slave-k6lz krb5kdc[8825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632470, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.66@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM
Sep 28 21:54:30 dist-test-slave-k6lz krb5kdc[8825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632470, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.66@KRBTEST.COM for oryx/127.2.182.124@KRBTEST.COM
I20260928 21:54:30.255808 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 9842
I20260928 21:54:30.255977 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 9200
I20260928 21:54:30.257743 8910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:30.246934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.66:48083 (local address 127.2.182.126:43551)
0928 21:54:30.246984 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:30.246985 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:30.246995 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:54:30.247467 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:30.247468 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:30.247481 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:30.247545 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:30.248373 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:30.249044 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:30.250299 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:30.250592 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:30.250657 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:30.253947 (+ 3290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:30.253949 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:30.253950 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:30.253959 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:30.255799 (+ 1840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:30.256598 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:30.256599 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:30.256600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:30.256667 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:30.256973 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:30.256974 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:30.256975 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:30.257119 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:30.257160 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:30.257555 (+ 395us) server_negotiation.cc:299] Negotiation successful
0928 21:54:30.257650 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 21:54:30.257745 9840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:30.246561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.66:58645 (local address 127.2.182.124:40387)
0928 21:54:30.246989 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:30.246991 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:30.247001 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:54:30.247597 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:30.247599 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:30.247611 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:30.247685 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:30.248624 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:30.249276 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:30.250376 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:30.250670 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:30.250725 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:30.254417 (+ 3692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:30.254419 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:30.254419 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:30.254428 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:30.255814 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:30.256837 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:30.256838 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:30.256838 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:30.256899 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:30.257092 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:30.257093 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:30.257093 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:30.257232 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:30.257272 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:30.257556 (+ 284us) server_negotiation.cc:299] Negotiation successful
0928 21:54:30.257648 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403}
I20260928 21:54:30.257752 9519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:30.246894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.66:41105 (local address 127.2.182.125:33661)
0928 21:54:30.246938 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:30.246940 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:30.247126 (+ 186us) server_negotiation.cc:407] Connection header received
0928 21:54:30.247602 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:30.247604 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:30.247613 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:30.247662 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:30.248327 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:30.248927 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:30.250414 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:30.250612 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:30.250665 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:30.253640 (+ 2975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:30.253642 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:30.253642 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:30.253651 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:30.255817 (+ 2166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:30.256479 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:30.256481 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:30.256481 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:30.256535 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:30.256977 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:30.256977 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:30.256978 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:30.257120 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:30.257161 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:30.257557 (+ 396us) server_negotiation.cc:299] Negotiation successful
0928 21:54:30.257646 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 21:54:30.258404 9977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:30.246805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.125:33661 (local address 127.2.182.66:41105)
0928 21:54:30.247054 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:30.247063 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:30.247368 (+ 305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:30.247693 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:30.247695 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:30.248013 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:30.248223 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:30.248228 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:30.249027 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:30.249028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:30.250297 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:30.250299 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:30.250717 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:30.250718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:30.250840 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:30.251561 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:30.251565 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:30.253540 (+ 1975us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:30.255904 (+ 2364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:30.255919 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:30.255922 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:30.256300 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:30.256847 (+ 547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:30.256848 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:30.256848 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:30.256907 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:30.257220 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:30.257221 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:30.257396 (+ 175us) client_negotiation.cc:769] Sending connection context
0928 21:54:30.257873 (+ 477us) client_negotiation.cc:240] Negotiation successful
0928 21:54:30.258195 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":239,"thread_start_us":61,"threads_started":1}
I20260928 21:54:30.258402 9975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:30.246516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.124:40387 (local address 127.2.182.66:58645)
0928 21:54:30.246692 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:30.246741 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:30.247465 (+ 724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:30.248327 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:30.248328 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:30.248482 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:30.248535 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:30.248536 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:30.249324 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:30.249325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:30.250299 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:30.250300 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:30.250738 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:30.250739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:30.250838 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:30.251534 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:30.251545 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:30.254325 (+ 2780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:30.255961 (+ 1636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:30.255962 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:30.255963 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:30.256662 (+ 699us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:30.256965 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:30.256966 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:30.256966 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:30.257018 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:30.257312 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:30.257313 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:30.257396 (+ 83us) client_negotiation.cc:769] Sending connection context
0928 21:54:30.257873 (+ 477us) client_negotiation.cc:240] Negotiation successful
0928 21:54:30.258128 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":10,"thread_start_us":88,"threads_started":1}
I20260928 21:54:30.259022 9976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:30.246516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:43551 (local address 127.2.182.66:48083)
0928 21:54:30.246764 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:30.246774 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:30.247368 (+ 594us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:30.247627 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:30.247631 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:30.248203 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:30.248259 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:30.248261 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:30.249123 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:30.249124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:30.250204 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:30.250207 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:30.250678 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:30.250679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:30.250840 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:30.251641 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:30.251645 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:30.253838 (+ 2193us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:30.255904 (+ 2066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:30.255919 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:30.255922 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:30.256520 (+ 598us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:30.256745 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:30.256745 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:30.256746 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:30.256901 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:30.257201 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:30.257207 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:30.257396 (+ 189us) client_negotiation.cc:769] Sending connection context
0928 21:54:30.258785 (+ 1389us) client_negotiation.cc:240] Negotiation successful
0928 21:54:30.258912 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":207,"spinlock_wait_cycles":8448,"thread_start_us":89,"threads_started":1}
I20260928 21:54:30.261700 9973 heartbeater.cc:347] Connected to a master server at 127.2.182.125:33661
I20260928 21:54:30.261826 9973 heartbeater.cc:464] Registering TS with master...
I20260928 21:54:30.262032 9973 heartbeater.cc:511] Master 127.2.182.125:33661 requested a full tablet report, sending...
I20260928 21:54:30.262475 9972 heartbeater.cc:347] Connected to a master server at 127.2.182.126:43551
I20260928 21:54:30.262547 9972 heartbeater.cc:464] Registering TS with master...
I20260928 21:54:30.262475 9971 heartbeater.cc:347] Connected to a master server at 127.2.182.124:40387
I20260928 21:54:30.262560 9468 ts_manager.cc:194] Registered new tserver with Master: d6cf3fea18684e48bc758ab6bf07563e (127.2.182.66:33051)
I20260928 21:54:30.262603 9971 heartbeater.cc:464] Registering TS with master...
I20260928 21:54:30.262625 9972 heartbeater.cc:511] Master 127.2.182.126:43551 requested a full tablet report, sending...
I20260928 21:54:30.262647 9971 heartbeater.cc:511] Master 127.2.182.124:40387 requested a full tablet report, sending...
I20260928 21:54:30.262717 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.67:33249
--local_ip_for_outbound_sockets=127.2.182.67
--tserver_master_addrs=127.2.182.126:43551
--webserver_port=43437
--webserver_interface=127.2.182.67
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/krb5kdc/oryx_127.2.182.67.keytab
--principal=oryx/127.2.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:45923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.182.126:43551,127.2.182.125:33661
--tserver_master_addrs=127.2.182.126:43551,127.2.182.125:33661,127.2.182.124:40387 with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:54:30.262862 8870 ts_manager.cc:194] Re-registered known tserver with Master: d6cf3fea18684e48bc758ab6bf07563e (127.2.182.66:33051)
I20260928 21:54:30.262902 9655 ts_manager.cc:194] Registered new tserver with Master: d6cf3fea18684e48bc758ab6bf07563e (127.2.182.66:33051)
I20260928 21:54:30.263317 8870 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.182.66@KRBTEST.COM'} at 127.2.182.66:48083
W20260928 21:54:30.344027 9978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:54:30.344250 9978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:54:30.344275 9978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:54:30.345489 9978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:54:30.345547 9978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:54:30.345587 9978 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.67
I20260928 21:54:30.347193 9978 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:45923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/krb5kdc/oryx_127.2.182.67.keytab
--principal=oryx/127.2.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.67:33249
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.182.67
--webserver_port=43437
--tserver_master_addrs=127.2.182.126:43551,127.2.182.125:33661,127.2.182.124:40387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.67
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:54:30.347477 9978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:54:30.347724 9978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:54:30.353147 9986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:30.353374 9983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:30.353377 9984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:30.354449 9978 server_base.cc:1034] running on GCE node
I20260928 21:54:30.354682 9978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:30.354991 9978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:30.356160 9978 hybrid_clock.cc:648] HybridClock initialized: now 1790632470356141 us; error 30 us; skew 500 ppm
Sep 28 21:54:30 dist-test-slave-k6lz krb5kdc[8825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632470, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:54:30.358759 9978 init.cc:377] Logged in from keytab as oryx/127.2.182.67@KRBTEST.COM (short username oryx)
I20260928 21:54:30.359218 9978 webserver.cc:460] Webserver started at http://127.2.182.67:43437/ using document root <none> and password file <none>
I20260928 21:54:30.359449 9978 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:30.359509 9978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:30.360972 9978 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:54:30.361640 9993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:30.361938 9978 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:54:30.362015 9978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-2/wal
uuid: "65646a62d4364636adf7ec56574214c0"
format_stamp: "Formatted at 2026-09-28 21:54:29 on dist-test-slave-k6lz"
server_key: "22820945eb66ada5ac565fc871aeb7f5"
server_key_iv: "5023178be24e82bb02cec421d0674fa7"
server_key_version: "encryptionkey@0"
I20260928 21:54:30.362411 9978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:30.373361 9978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:54:30.375540 9978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:54:30.375660 9978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:54:30.375963 9978 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:54:30.376262 9978 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:54:30.376294 9978 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:30.376315 9978 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:54:30.376330 9978 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:30.381803 9978 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.67:33249
I20260928 21:54:30.381868 10106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.67:33249 every 8 connection(s)
I20260928 21:54:30.382257 9978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
I20260928 21:54:30.388089 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 9978
Sep 28 21:54:30 dist-test-slave-k6lz krb5kdc[8825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632470, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.67@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM
Sep 28 21:54:30 dist-test-slave-k6lz krb5kdc[8825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632470, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.67@KRBTEST.COM for oryx/127.2.182.124@KRBTEST.COM
Sep 28 21:54:30 dist-test-slave-k6lz krb5kdc[8825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632470, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.67@KRBTEST.COM for oryx/127.2.182.125@KRBTEST.COM
I20260928 21:54:30.396430 10112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:30.383020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:43551 (local address 127.2.182.67:57871)
0928 21:54:30.383226 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:30.383277 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:30.383873 (+ 596us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:30.384174 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:30.384178 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:30.384467 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:30.384660 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:30.384664 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:30.386029 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:30.386030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:30.386759 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:30.386760 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:30.387176 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:30.387177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:30.387249 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:30.387921 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:30.387930 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:30.391556 (+ 3626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:30.394595 (+ 3039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:30.394606 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:30.394610 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:30.394932 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:30.395153 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:30.395154 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:30.395154 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:30.395320 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:30.395638 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:30.395645 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:30.395849 (+ 204us) client_negotiation.cc:769] Sending connection context
0928 21:54:30.395968 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 21:54:30.396162 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":1,"spinlock_wait_cycles":15744,"thread_start_us":109,"threads_started":1}
I20260928 21:54:30.396888 10113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:30.383464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.125:33661 (local address 127.2.182.67:41937)
0928 21:54:30.383566 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:30.383575 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:30.383919 (+ 344us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:30.384539 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:30.384548 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:30.384676 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:30.384723 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:30.384724 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:30.385434 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:30.385434 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:30.386573 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:30.386575 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:30.387091 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:30.387091 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:30.387231 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:30.388981 (+ 1750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:30.388986 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:30.390963 (+ 1977us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:30.395848 (+ 4885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:30.395849 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:30.395850 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:30.396060 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:30.396251 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:30.396251 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:30.396252 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:30.396304 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:30.396570 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:30.396570 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:30.396611 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 21:54:30.396667 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 21:54:30.396783 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"mutex_wait_us":2,"thread_start_us":58,"threads_started":1}
I20260928 21:54:30.396893 8910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:30.383299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.67:57871 (local address 127.2.182.126:43551)
0928 21:54:30.383371 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:30.383373 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:30.383384 (+ 11us) server_negotiation.cc:407] Connection header received
0928 21:54:30.383994 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:30.383995 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:30.384008 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:30.384037 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:30.384754 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:30.385537 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:30.386835 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:30.387121 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:30.387175 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:30.391636 (+ 4461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:30.391637 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:30.391638 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:30.391644 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:30.394503 (+ 2859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:30.395015 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:30.395017 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:30.395017 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:30.395085 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:30.395412 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:30.395413 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:30.395414 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:30.395557 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:30.395613 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:30.396701 (+ 1088us) server_negotiation.cc:299] Negotiation successful
0928 21:54:30.396787 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 21:54:30.397215 9519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:30.383549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.67:41937 (local address 127.2.182.125:33661)
0928 21:54:30.383594 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:30.383595 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:30.383608 (+ 13us) server_negotiation.cc:407] Connection header received
0928 21:54:30.384048 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:30.384050 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:30.384060 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:30.384109 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:30.384813 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:30.385321 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:30.386673 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:30.386999 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:30.387068 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:30.391053 (+ 3985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:30.391055 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:30.391055 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:30.391063 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:30.395658 (+ 4595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:30.396134 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:30.396135 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:30.396135 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:30.396190 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:30.396369 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:30.396369 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:30.396370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:30.396487 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:30.396545 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:30.397058 (+ 513us) server_negotiation.cc:299] Negotiation successful
0928 21:54:30.397133 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 21:54:30.397219 10108 heartbeater.cc:347] Connected to a master server at 127.2.182.126:43551
I20260928 21:54:30.397318 10108 heartbeater.cc:464] Registering TS with master...
I20260928 21:54:30.397473 10109 heartbeater.cc:347] Connected to a master server at 127.2.182.125:33661
I20260928 21:54:30.397506 10109 heartbeater.cc:464] Registering TS with master...
I20260928 21:54:30.397526 10108 heartbeater.cc:511] Master 127.2.182.126:43551 requested a full tablet report, sending...
I20260928 21:54:30.397552 10109 heartbeater.cc:511] Master 127.2.182.125:33661 requested a full tablet report, sending...
I20260928 21:54:30.397882 8870 ts_manager.cc:194] Re-registered known tserver with Master: 65646a62d4364636adf7ec56574214c0 (127.2.182.67:33249)
I20260928 21:54:30.397944 9468 ts_manager.cc:194] Registered new tserver with Master: 65646a62d4364636adf7ec56574214c0 (127.2.182.67:33249)
I20260928 21:54:30.398360 8870 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.182.67@KRBTEST.COM'} at 127.2.182.67:57871
I20260928 21:54:30.398633 10111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:30.383019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.124:40387 (local address 127.2.182.67:60587)
0928 21:54:30.383226 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:30.383277 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:30.383881 (+ 604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:30.384192 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:30.384193 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:30.384547 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:30.384661 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:30.384667 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:30.385611 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:30.385612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:30.386569 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:30.386571 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:30.387173 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:30.387174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:30.387237 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:30.387948 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:30.387953 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:30.390964 (+ 3011us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:30.396022 (+ 5058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:30.396023 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:30.396023 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:30.397561 (+ 1538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:30.397995 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:30.397995 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:30.397996 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:30.398050 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:30.398315 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:30.398316 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:30.398369 (+ 53us) client_negotiation.cc:769] Sending connection context
0928 21:54:30.398431 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 21:54:30.398533 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":70,"thread_start_us":93,"threads_started":1}
I20260928 21:54:30.398731 10121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:30.389473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55114 (local address 127.2.182.124:40387)
0928 21:54:30.389921 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:30.389923 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:30.392006 (+ 2083us) server_negotiation.cc:407] Connection header received
0928 21:54:30.392026 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:30.392027 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:30.392037 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:30.392083 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:30.392723 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:30.393409 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:30.394321 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:30.394608 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:30.394663 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:30.395915 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:30.395916 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:30.395916 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:30.395922 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:30.397474 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:30.397822 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:30.397823 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:30.397823 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:30.397914 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:30.398100 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:30.398101 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:30.398102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:30.398224 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:30.398312 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:30.398561 (+ 249us) server_negotiation.cc:299] Negotiation successful
0928 21:54:30.398636 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":51,"threads_started":1}
I20260928 21:54:30.399385 9840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:30.383002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.67:60587 (local address 127.2.182.124:40387)
0928 21:54:30.383048 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:30.383050 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:30.383381 (+ 331us) server_negotiation.cc:407] Connection header received
0928 21:54:30.383994 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:30.383995 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:30.384008 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:30.384058 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:30.384774 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:30.385537 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:30.386844 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:30.387034 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:30.387090 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:30.391105 (+ 4015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:30.391107 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:30.391107 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:30.391114 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:30.395908 (+ 4794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:30.397846 (+ 1938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:30.397847 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:30.397847 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:30.397930 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:30.398154 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:30.398155 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:30.398155 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:30.398247 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:30.398291 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:30.399190 (+ 899us) server_negotiation.cc:299] Negotiation successful
0928 21:54:30.399281 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 21:54:30.400440 10107 heartbeater.cc:347] Connected to a master server at 127.2.182.124:40387
I20260928 21:54:30.400494 10107 heartbeater.cc:464] Registering TS with master...
I20260928 21:54:30.400560 10107 heartbeater.cc:511] Master 127.2.182.124:40387 requested a full tablet report, sending...
I20260928 21:54:30.400759 10119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:30.389229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37060 (local address 127.2.182.125:33661)
0928 21:54:30.389358 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:30.389359 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:30.389844 (+ 485us) server_negotiation.cc:407] Connection header received
0928 21:54:30.389888 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:30.389889 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:30.389901 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:30.389961 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:30.390686 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:30.391435 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:30.392450 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:30.392743 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:30.392778 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:30.394108 (+ 1330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:30.394109 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:30.394110 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:30.394116 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:30.396773 (+ 2657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:30.399884 (+ 3111us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:30.399885 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:30.399886 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:30.399965 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:30.400193 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:30.400194 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:30.400194 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:30.400306 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:30.400404 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:30.400506 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 21:54:30.400664 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":116,"threads_started":1}
I20260928 21:54:30.400840 9655 ts_manager.cc:194] Registered new tserver with Master: 65646a62d4364636adf7ec56574214c0 (127.2.182.67:33249)
I20260928 21:54:30.401628 10123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:30.390531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46408 (local address 127.2.182.126:43551)
0928 21:54:30.390708 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:30.390710 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:30.390719 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:54:30.390741 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:30.390742 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:30.390753 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:30.390914 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:30.391611 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:30.392348 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:30.393326 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:30.393609 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:30.393685 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:30.398810 (+ 5125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:30.398812 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:30.398812 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:30.398819 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:30.400492 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:30.400887 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:30.400888 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:30.400888 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:30.400944 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:30.401132 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:30.401133 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:30.401133 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:30.401254 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:30.401339 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:30.401460 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 21:54:30.401541 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":87,"threads_started":1}
I20260928 21:54:30.403416 8870 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46408:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:54:30.405092 8870 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:54:30.412940 10130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:30.409347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56200 (local address 127.2.182.65:39731)
0928 21:54:30.409838 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:30.409843 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:30.409851 (+ 8us) server_negotiation.cc:407] Connection header received
0928 21:54:30.409968 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:30.409970 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:30.409986 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:30.410058 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:54:30.410161 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:30.410939 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:30.412004 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:30.412586 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:30.412664 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:30.412745 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:30.412807 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 21:54:30.412848 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":87,"threads_started":1}
I20260928 21:54:30.412930 9428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:30.409200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.65:39731 (local address 127.0.0.1:56200)
0928 21:54:30.409359 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:30.409369 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:30.409411 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:30.410068 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:30.410069 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:30.410070 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:54:30.410095 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:30.410096 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:30.411010 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:30.411011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:30.411930 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:30.411933 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:30.412673 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:30.412674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:30.412733 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:30.412737 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:54:30.412792 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 21:54:30.412847 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":83,"threads_started":1}
I20260928 21:54:30.413856 10129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:30.409238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.66:33051 (local address 127.0.0.1:34384)
0928 21:54:30.409673 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:30.409682 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:30.409732 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:30.409915 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:30.409916 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:30.409920 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:54:30.409974 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:30.409976 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:30.411624 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:30.411624 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:30.412446 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:30.412449 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:30.413620 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:30.413621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:30.413700 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:30.413705 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 21:54:30.413738 (+ 33us) client_negotiation.cc:240] Negotiation successful
0928 21:54:30.413781 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"mutex_wait_us":22,"thread_start_us":85,"threads_started":1}
I20260928 21:54:30.413839 9768 tablet_service.cc:1467] Processing CreateTablet for tablet e44118a0519846ec81d8f2d74750c6fb (DEFAULT_TABLE table=test-table [id=ef9e482a5acf4fa1ac711ba139bc2cb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:54:30.414192 9768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e44118a0519846ec81d8f2d74750c6fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:54:30.414191 10132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:30.409521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34384 (local address 127.2.182.66:33051)
0928 21:54:30.409668 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:30.409673 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:30.409723 (+ 50us) server_negotiation.cc:407] Connection header received
0928 21:54:30.409795 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:30.409798 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:30.409814 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:30.409893 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:54:30.410040 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:30.411531 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:30.412535 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:30.413529 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:30.413964 (+ 435us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:30.414052 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:30.414077 (+ 25us) server_negotiation.cc:299] Negotiation successful
0928 21:54:30.414115 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":67,"threads_started":1}
I20260928 21:54:30.414866 10128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:30.409146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.67:33249 (local address 127.0.0.1:49128)
0928 21:54:30.409344 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:30.409357 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:30.409411 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:30.410313 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:30.410313 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:30.410315 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:54:30.410366 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:30.410367 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:30.411619 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:30.411620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:30.413715 (+ 2095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:30.413717 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:30.414638 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:30.414639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:30.414710 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:30.414714 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:54:30.414760 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 21:54:30.414802 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":41}
I20260928 21:54:30.414987 10131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:30.409325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49128 (local address 127.2.182.67:33249)
0928 21:54:30.410168 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:30.410172 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:30.410181 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:54:30.410213 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:30.410215 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:30.410229 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:30.410308 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:54:30.410430 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:30.411529 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:30.413797 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:30.414554 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:30.414617 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:30.414689 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:30.414772 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 21:54:30.414805 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":805,"thread_start_us":119,"threads_started":1}
I20260928 21:54:30.414840 9905 tablet_service.cc:1467] Processing CreateTablet for tablet e44118a0519846ec81d8f2d74750c6fb (DEFAULT_TABLE table=test-table [id=ef9e482a5acf4fa1ac711ba139bc2cb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:54:30.415194 9905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e44118a0519846ec81d8f2d74750c6fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:54:30.415596 10041 tablet_service.cc:1467] Processing CreateTablet for tablet e44118a0519846ec81d8f2d74750c6fb (DEFAULT_TABLE table=test-table [id=ef9e482a5acf4fa1ac711ba139bc2cb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:54:30.415922 10041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e44118a0519846ec81d8f2d74750c6fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:54:30.418440 10133 tablet_bootstrap.cc:492] T e44118a0519846ec81d8f2d74750c6fb P 4438ab182bef4715bd7a150fab1403bd: Bootstrap starting.
I20260928 21:54:30.419065 10134 tablet_bootstrap.cc:492] T e44118a0519846ec81d8f2d74750c6fb P 65646a62d4364636adf7ec56574214c0: Bootstrap starting.
I20260928 21:54:30.419382 10133 tablet_bootstrap.cc:654] T e44118a0519846ec81d8f2d74750c6fb P 4438ab182bef4715bd7a150fab1403bd: Neither blocks nor log segments found. Creating new log.
I20260928 21:54:30.419456 10135 tablet_bootstrap.cc:492] T e44118a0519846ec81d8f2d74750c6fb P d6cf3fea18684e48bc758ab6bf07563e: Bootstrap starting.
I20260928 21:54:30.419726 10134 tablet_bootstrap.cc:654] T e44118a0519846ec81d8f2d74750c6fb P 65646a62d4364636adf7ec56574214c0: Neither blocks nor log segments found. Creating new log.
I20260928 21:54:30.419705 10133 log.cc:824] T e44118a0519846ec81d8f2d74750c6fb P 4438ab182bef4715bd7a150fab1403bd: Log is configured to *not* fsync() on all Append() calls
I20260928 21:54:30.420011 10135 tablet_bootstrap.cc:654] T e44118a0519846ec81d8f2d74750c6fb P d6cf3fea18684e48bc758ab6bf07563e: Neither blocks nor log segments found. Creating new log.
I20260928 21:54:30.420012 10134 log.cc:824] T e44118a0519846ec81d8f2d74750c6fb P 65646a62d4364636adf7ec56574214c0: Log is configured to *not* fsync() on all Append() calls
I20260928 21:54:30.420310 10135 log.cc:824] T e44118a0519846ec81d8f2d74750c6fb P d6cf3fea18684e48bc758ab6bf07563e: Log is configured to *not* fsync() on all Append() calls
I20260928 21:54:30.420622 10133 tablet_bootstrap.cc:492] T e44118a0519846ec81d8f2d74750c6fb P 4438ab182bef4715bd7a150fab1403bd: No bootstrap required, opened a new log
I20260928 21:54:30.420709 10133 ts_tablet_manager.cc:1397] T e44118a0519846ec81d8f2d74750c6fb P 4438ab182bef4715bd7a150fab1403bd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 21:54:30.421079 10135 tablet_bootstrap.cc:492] T e44118a0519846ec81d8f2d74750c6fb P d6cf3fea18684e48bc758ab6bf07563e: No bootstrap required, opened a new log
I20260928 21:54:30.421145 10135 ts_tablet_manager.cc:1397] T e44118a0519846ec81d8f2d74750c6fb P d6cf3fea18684e48bc758ab6bf07563e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:54:30.421783 10134 tablet_bootstrap.cc:492] T e44118a0519846ec81d8f2d74750c6fb P 65646a62d4364636adf7ec56574214c0: No bootstrap required, opened a new log
I20260928 21:54:30.421850 10134 ts_tablet_manager.cc:1397] T e44118a0519846ec81d8f2d74750c6fb P 65646a62d4364636adf7ec56574214c0: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260928 21:54:30.422834 10135 raft_consensus.cc:348] T e44118a0519846ec81d8f2d74750c6fb P d6cf3fea18684e48bc758ab6bf07563e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65646a62d4364636adf7ec56574214c0" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 33249 } } peers { permanent_uuid: "d6cf3fea18684e48bc758ab6bf07563e" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 33051 } } peers { permanent_uuid: "4438ab182bef4715bd7a150fab1403bd" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 39731 } }
I20260928 21:54:30.422971 10135 raft_consensus.cc:374] T e44118a0519846ec81d8f2d74750c6fb P d6cf3fea18684e48bc758ab6bf07563e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:54:30.423000 10135 raft_consensus.cc:729] T e44118a0519846ec81d8f2d74750c6fb P d6cf3fea18684e48bc758ab6bf07563e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6cf3fea18684e48bc758ab6bf07563e, State: Initialized, Role: FOLLOWER
I20260928 21:54:30.423135 10135 consensus_queue.cc:260] T e44118a0519846ec81d8f2d74750c6fb P d6cf3fea18684e48bc758ab6bf07563e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65646a62d4364636adf7ec56574214c0" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 33249 } } peers { permanent_uuid: "d6cf3fea18684e48bc758ab6bf07563e" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 33051 } } peers { permanent_uuid: "4438ab182bef4715bd7a150fab1403bd" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 39731 } }
I20260928 21:54:30.423096 10133 raft_consensus.cc:348] T e44118a0519846ec81d8f2d74750c6fb P 4438ab182bef4715bd7a150fab1403bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65646a62d4364636adf7ec56574214c0" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 33249 } } peers { permanent_uuid: "d6cf3fea18684e48bc758ab6bf07563e" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 33051 } } peers { permanent_uuid: "4438ab182bef4715bd7a150fab1403bd" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 39731 } }
I20260928 21:54:30.423255 10133 raft_consensus.cc:374] T e44118a0519846ec81d8f2d74750c6fb P 4438ab182bef4715bd7a150fab1403bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:54:30.423285 10133 raft_consensus.cc:729] T e44118a0519846ec81d8f2d74750c6fb P 4438ab182bef4715bd7a150fab1403bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4438ab182bef4715bd7a150fab1403bd, State: Initialized, Role: FOLLOWER
I20260928 21:54:30.423429 10135 ts_tablet_manager.cc:1428] T e44118a0519846ec81d8f2d74750c6fb P d6cf3fea18684e48bc758ab6bf07563e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 21:54:30.423412 10133 consensus_queue.cc:260] T e44118a0519846ec81d8f2d74750c6fb P 4438ab182bef4715bd7a150fab1403bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65646a62d4364636adf7ec56574214c0" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 33249 } } peers { permanent_uuid: "d6cf3fea18684e48bc758ab6bf07563e" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 33051 } } peers { permanent_uuid: "4438ab182bef4715bd7a150fab1403bd" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 39731 } }
I20260928 21:54:30.423636 10133 ts_tablet_manager.cc:1428] T e44118a0519846ec81d8f2d74750c6fb P 4438ab182bef4715bd7a150fab1403bd: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 21:54:30.423772 9835 heartbeater.cc:503] Master 127.2.182.126:43551 was elected leader, sending a full tablet report...
I20260928 21:54:30.423756 10134 raft_consensus.cc:348] T e44118a0519846ec81d8f2d74750c6fb P 65646a62d4364636adf7ec56574214c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65646a62d4364636adf7ec56574214c0" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 33249 } } peers { permanent_uuid: "d6cf3fea18684e48bc758ab6bf07563e" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 33051 } } peers { permanent_uuid: "4438ab182bef4715bd7a150fab1403bd" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 39731 } }
I20260928 21:54:30.423895 10134 raft_consensus.cc:374] T e44118a0519846ec81d8f2d74750c6fb P 65646a62d4364636adf7ec56574214c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:54:30.423926 10134 raft_consensus.cc:729] T e44118a0519846ec81d8f2d74750c6fb P 65646a62d4364636adf7ec56574214c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65646a62d4364636adf7ec56574214c0, State: Initialized, Role: FOLLOWER
I20260928 21:54:30.424019 10134 consensus_queue.cc:260] T e44118a0519846ec81d8f2d74750c6fb P 65646a62d4364636adf7ec56574214c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65646a62d4364636adf7ec56574214c0" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 33249 } } peers { permanent_uuid: "d6cf3fea18684e48bc758ab6bf07563e" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 33051 } } peers { permanent_uuid: "4438ab182bef4715bd7a150fab1403bd" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 39731 } }
I20260928 21:54:30.424242 10134 ts_tablet_manager.cc:1428] T e44118a0519846ec81d8f2d74750c6fb P 65646a62d4364636adf7ec56574214c0: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260928 21:54:30.424654 10108 heartbeater.cc:503] Master 127.2.182.126:43551 was elected leader, sending a full tablet report...
I20260928 21:54:30.424808 9972 heartbeater.cc:503] Master 127.2.182.126:43551 was elected leader, sending a full tablet report...
I20260928 21:54:30.494241 9840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:30.491444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55128 (local address 127.2.182.124:40387)
0928 21:54:30.491504 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:30.491506 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:30.491515 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:54:30.491540 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:30.491541 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:30.491559 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:30.491629 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:54:30.491978 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:30.492470 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:30.493424 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:30.493897 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:30.493968 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:30.494030 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:30.494149 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 21:54:30.494179 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 21:54:30.494242 10128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:30.491367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.124:40387 (local address 127.0.0.1:55128)
0928 21:54:30.491414 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:30.491426 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:30.491478 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:30.491668 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:30.491670 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:30.491672 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:54:30.491810 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:30.491813 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:30.492567 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:30.492568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:30.493319 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:30.493321 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:30.494012 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:30.494013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:30.494076 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:30.494079 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:54:30.494134 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 21:54:30.494179 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260928 21:54:30.494715 9665 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 8e56bb6247ad4b39a8bf4340451035a5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260928 21:54:30.495620 9694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e56bb6247ad4b39a8bf4340451035a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d22a18a4a71545768146dab94c9a04a1. Latest consensus state: current_term: 1 leader_uuid: "d22a18a4a71545768146dab94c9a04a1" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d22a18a4a71545768146dab94c9a04a1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43551 } } peers { permanent_uuid: "59256c4c13d94676ac3e3b7e1e51a6a8" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 33661 } attrs { promote: false } } peers { permanent_uuid: "8e56bb6247ad4b39a8bf4340451035a5" member_type: NON_VOTER last_known_addr { host: "127.2.182.124" port: 40387 } attrs { promote: true } } }
I20260928 21:54:30.495729 9694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e56bb6247ad4b39a8bf4340451035a5 [sys.catalog]: This master's current role is: LEARNER
W20260928 21:54:30.496778 9974 tablet.cc:2369] T e44118a0519846ec81d8f2d74750c6fb P d6cf3fea18684e48bc758ab6bf07563e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:54:30.577454 10141 raft_consensus.cc:482] T e44118a0519846ec81d8f2d74750c6fb P 65646a62d4364636adf7ec56574214c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:54:30.577582 10141 raft_consensus.cc:504] T e44118a0519846ec81d8f2d74750c6fb P 65646a62d4364636adf7ec56574214c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65646a62d4364636adf7ec56574214c0" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 33249 } } peers { permanent_uuid: "d6cf3fea18684e48bc758ab6bf07563e" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 33051 } } peers { permanent_uuid: "4438ab182bef4715bd7a150fab1403bd" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 39731 } }
I20260928 21:54:30.578102 10141 leader_election.cc:288] T e44118a0519846ec81d8f2d74750c6fb P 65646a62d4364636adf7ec56574214c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6cf3fea18684e48bc758ab6bf07563e (127.2.182.66:33051), 4438ab182bef4715bd7a150fab1403bd (127.2.182.65:39731)
I20260928 21:54:30.580965 10132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:30.578355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.67:55671 (local address 127.2.182.66:33051)
0928 21:54:30.578419 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:30.578421 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:30.578471 (+ 50us) server_negotiation.cc:407] Connection header received
0928 21:54:30.578496 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:30.578497 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:30.578505 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:30.578550 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:54:30.578794 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:30.579288 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:30.580279 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:30.580654 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:30.580719 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:30.580729 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:30.580861 (+ 132us) server_negotiation.cc:299] Negotiation successful
0928 21:54:30.580901 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 21:54:30.580994 10111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:30.578295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.66:33051 (local address 127.2.182.67:55671)
0928 21:54:30.578403 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:30.578412 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:30.578437 (+ 25us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:30.578569 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:30.578570 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:30.578572 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:54:30.578636 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:30.578638 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:30.579403 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:30.579404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:30.580186 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:30.580188 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:30.580728 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:30.580729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:30.580796 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:30.580799 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:54:30.580843 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 21:54:30.580891 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95}
I20260928 21:54:30.580993 10130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:30.578492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.67:41639 (local address 127.2.182.65:39731)
0928 21:54:30.578533 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:30.578535 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:30.578543 (+ 8us) server_negotiation.cc:407] Connection header received
0928 21:54:30.578566 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:30.578567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:30.578574 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:30.578619 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:54:30.578775 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:30.579249 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:30.580285 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:30.580655 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:30.580726 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:30.580735 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:30.580884 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 21:54:30.580925 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 21:54:30.581010 10113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:30.578279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.65:39731 (local address 127.2.182.67:41639)
0928 21:54:30.578349 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:30.578360 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:30.578405 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:30.578637 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:30.578638 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:30.578640 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:54:30.578700 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:30.578702 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:30.579322 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:30.579323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:30.580195 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:30.580198 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:30.580741 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:30.580742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:30.580822 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:30.580825 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:54:30.580869 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 21:54:30.580911 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":1}
I20260928 21:54:30.581305 9788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e44118a0519846ec81d8f2d74750c6fb" candidate_uuid: "65646a62d4364636adf7ec56574214c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4438ab182bef4715bd7a150fab1403bd" is_pre_election: true
I20260928 21:54:30.581328 9925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e44118a0519846ec81d8f2d74750c6fb" candidate_uuid: "65646a62d4364636adf7ec56574214c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6cf3fea18684e48bc758ab6bf07563e" is_pre_election: true
I20260928 21:54:30.581458 9788 raft_consensus.cc:2454] T e44118a0519846ec81d8f2d74750c6fb P 4438ab182bef4715bd7a150fab1403bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65646a62d4364636adf7ec56574214c0 in term 0.
I20260928 21:54:30.581460 9925 raft_consensus.cc:2454] T e44118a0519846ec81d8f2d74750c6fb P d6cf3fea18684e48bc758ab6bf07563e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65646a62d4364636adf7ec56574214c0 in term 0.
I20260928 21:54:30.581692 9995 leader_election.cc:302] T e44118a0519846ec81d8f2d74750c6fb P 65646a62d4364636adf7ec56574214c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4438ab182bef4715bd7a150fab1403bd, 65646a62d4364636adf7ec56574214c0; no voters:
I20260928 21:54:30.581871 10141 raft_consensus.cc:2781] T e44118a0519846ec81d8f2d74750c6fb P 65646a62d4364636adf7ec56574214c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:54:30.581931 10141 raft_consensus.cc:482] T e44118a0519846ec81d8f2d74750c6fb P 65646a62d4364636adf7ec56574214c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:54:30.581954 10141 raft_consensus.cc:3037] T e44118a0519846ec81d8f2d74750c6fb P 65646a62d4364636adf7ec56574214c0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:54:30.582720 10141 raft_consensus.cc:504] T e44118a0519846ec81d8f2d74750c6fb P 65646a62d4364636adf7ec56574214c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65646a62d4364636adf7ec56574214c0" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 33249 } } peers { permanent_uuid: "d6cf3fea18684e48bc758ab6bf07563e" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 33051 } } peers { permanent_uuid: "4438ab182bef4715bd7a150fab1403bd" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 39731 } }
I20260928 21:54:30.582926 10141 leader_election.cc:288] T e44118a0519846ec81d8f2d74750c6fb P 65646a62d4364636adf7ec56574214c0 [CANDIDATE]: Term 1 election: Requested vote from peers d6cf3fea18684e48bc758ab6bf07563e (127.2.182.66:33051), 4438ab182bef4715bd7a150fab1403bd (127.2.182.65:39731)
I20260928 21:54:30.583047 9925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e44118a0519846ec81d8f2d74750c6fb" candidate_uuid: "65646a62d4364636adf7ec56574214c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6cf3fea18684e48bc758ab6bf07563e"
I20260928 21:54:30.583125 9925 raft_consensus.cc:3037] T e44118a0519846ec81d8f2d74750c6fb P d6cf3fea18684e48bc758ab6bf07563e [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:54:30.583155 9788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e44118a0519846ec81d8f2d74750c6fb" candidate_uuid: "65646a62d4364636adf7ec56574214c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4438ab182bef4715bd7a150fab1403bd"
I20260928 21:54:30.583249 9788 raft_consensus.cc:3037] T e44118a0519846ec81d8f2d74750c6fb P 4438ab182bef4715bd7a150fab1403bd [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:54:30.583876 9925 raft_consensus.cc:2454] T e44118a0519846ec81d8f2d74750c6fb P d6cf3fea18684e48bc758ab6bf07563e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65646a62d4364636adf7ec56574214c0 in term 1.
I20260928 21:54:30.584003 9788 raft_consensus.cc:2454] T e44118a0519846ec81d8f2d74750c6fb P 4438ab182bef4715bd7a150fab1403bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65646a62d4364636adf7ec56574214c0 in term 1.
I20260928 21:54:30.584062 9994 leader_election.cc:302] T e44118a0519846ec81d8f2d74750c6fb P 65646a62d4364636adf7ec56574214c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65646a62d4364636adf7ec56574214c0, d6cf3fea18684e48bc758ab6bf07563e; no voters:
I20260928 21:54:30.584152 10141 raft_consensus.cc:2781] T e44118a0519846ec81d8f2d74750c6fb P 65646a62d4364636adf7ec56574214c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:54:30.584285 10141 raft_consensus.cc:686] T e44118a0519846ec81d8f2d74750c6fb P 65646a62d4364636adf7ec56574214c0 [term 1 LEADER]: Becoming Leader. State: Replica: 65646a62d4364636adf7ec56574214c0, State: Running, Role: LEADER
I20260928 21:54:30.584375 10141 consensus_queue.cc:237] T e44118a0519846ec81d8f2d74750c6fb P 65646a62d4364636adf7ec56574214c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65646a62d4364636adf7ec56574214c0" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 33249 } } peers { permanent_uuid: "d6cf3fea18684e48bc758ab6bf07563e" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 33051 } } peers { permanent_uuid: "4438ab182bef4715bd7a150fab1403bd" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 39731 } }
I20260928 21:54:30.585218 8870 catalog_manager.cc:5495] T e44118a0519846ec81d8f2d74750c6fb P 65646a62d4364636adf7ec56574214c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 65646a62d4364636adf7ec56574214c0 (127.2.182.67). New cstate: current_term: 1 leader_uuid: "65646a62d4364636adf7ec56574214c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65646a62d4364636adf7ec56574214c0" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 33249 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6cf3fea18684e48bc758ab6bf07563e" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4438ab182bef4715bd7a150fab1403bd" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 39731 } health_report { overall_health: UNKNOWN } } }
I20260928 21:54:30.585813 9430 sys_catalog.cc:455] T 00000000000000000000000000000000 P d22a18a4a71545768146dab94c9a04a1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d22a18a4a71545768146dab94c9a04a1" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d22a18a4a71545768146dab94c9a04a1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43551 } } peers { permanent_uuid: "59256c4c13d94676ac3e3b7e1e51a6a8" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 33661 } attrs { promote: false } } peers { permanent_uuid: "8e56bb6247ad4b39a8bf4340451035a5" member_type: NON_VOTER last_known_addr { host: "127.2.182.124" port: 40387 } attrs { promote: true } } }
I20260928 21:54:30.585903 9430 sys_catalog.cc:458] T 00000000000000000000000000000000 P d22a18a4a71545768146dab94c9a04a1 [sys.catalog]: This master's current role is: LEADER
I20260928 21:54:30.586387 9430 raft_consensus.cc:1050] T 00000000000000000000000000000000 P d22a18a4a71545768146dab94c9a04a1: attempting to promote NON_VOTER 8e56bb6247ad4b39a8bf4340451035a5 to VOTER
I20260928 21:54:30.586504 9430 consensus_queue.cc:237] T 00000000000000000000000000000000 P d22a18a4a71545768146dab94c9a04a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d22a18a4a71545768146dab94c9a04a1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43551 } } peers { permanent_uuid: "59256c4c13d94676ac3e3b7e1e51a6a8" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 33661 } attrs { promote: false } } peers { permanent_uuid: "8e56bb6247ad4b39a8bf4340451035a5" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 40387 } attrs { promote: false } }
I20260928 21:54:30.586880 9478 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 59256c4c13d94676ac3e3b7e1e51a6a8 [term 1 FOLLOWER]: Refusing update from remote peer d22a18a4a71545768146dab94c9a04a1: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260928 21:54:30.586885 9665 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8e56bb6247ad4b39a8bf4340451035a5 [term 1 LEARNER]: Refusing update from remote peer d22a18a4a71545768146dab94c9a04a1: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260928 21:54:30.587081 10149 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d22a18a4a71545768146dab94c9a04a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e56bb6247ad4b39a8bf4340451035a5" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 40387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260928 21:54:30.587253 9619 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d22a18a4a71545768146dab94c9a04a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59256c4c13d94676ac3e3b7e1e51a6a8" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 33661 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260928 21:54:30.587836 9619 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d22a18a4a71545768146dab94c9a04a1 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 8e56bb6247ad4b39a8bf4340451035a5 (127.2.182.124) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d22a18a4a71545768146dab94c9a04a1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43551 } } peers { permanent_uuid: "59256c4c13d94676ac3e3b7e1e51a6a8" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 33661 } attrs { promote: false } } peers { permanent_uuid: "8e56bb6247ad4b39a8bf4340451035a5" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 40387 } attrs { promote: false } } }
I20260928 21:54:30.588636 9619 sys_catalog.cc:455] T 00000000000000000000000000000000 P d22a18a4a71545768146dab94c9a04a1 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d22a18a4a71545768146dab94c9a04a1" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d22a18a4a71545768146dab94c9a04a1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43551 } } peers { permanent_uuid: "59256c4c13d94676ac3e3b7e1e51a6a8" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 33661 } attrs { promote: false } } peers { permanent_uuid: "8e56bb6247ad4b39a8bf4340451035a5" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 40387 } attrs { promote: false } } }
I20260928 21:54:30.588673 10149 sys_catalog.cc:455] T 00000000000000000000000000000000 P d22a18a4a71545768146dab94c9a04a1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d22a18a4a71545768146dab94c9a04a1" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d22a18a4a71545768146dab94c9a04a1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43551 } } peers { permanent_uuid: "59256c4c13d94676ac3e3b7e1e51a6a8" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 33661 } attrs { promote: false } } peers { permanent_uuid: "8e56bb6247ad4b39a8bf4340451035a5" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 40387 } attrs { promote: false } } }
I20260928 21:54:30.588723 10149 sys_catalog.cc:458] T 00000000000000000000000000000000 P d22a18a4a71545768146dab94c9a04a1 [sys.catalog]: This master's current role is: LEADER
I20260928 21:54:30.588737 9619 sys_catalog.cc:458] T 00000000000000000000000000000000 P d22a18a4a71545768146dab94c9a04a1 [sys.catalog]: This master's current role is: LEADER
I20260928 21:54:30.588917 9478 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 59256c4c13d94676ac3e3b7e1e51a6a8 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 8e56bb6247ad4b39a8bf4340451035a5 (127.2.182.124) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d22a18a4a71545768146dab94c9a04a1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43551 } } peers { permanent_uuid: "59256c4c13d94676ac3e3b7e1e51a6a8" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 33661 } attrs { promote: false } } peers { permanent_uuid: "8e56bb6247ad4b39a8bf4340451035a5" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 40387 } attrs { promote: false } } }
I20260928 21:54:30.589044 10147 sys_catalog.cc:455] T 00000000000000000000000000000000 P d22a18a4a71545768146dab94c9a04a1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d22a18a4a71545768146dab94c9a04a1" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d22a18a4a71545768146dab94c9a04a1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43551 } } peers { permanent_uuid: "59256c4c13d94676ac3e3b7e1e51a6a8" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 33661 } attrs { promote: false } } peers { permanent_uuid: "8e56bb6247ad4b39a8bf4340451035a5" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 40387 } attrs { promote: false } } }
I20260928 21:54:30.589111 10147 sys_catalog.cc:458] T 00000000000000000000000000000000 P d22a18a4a71545768146dab94c9a04a1 [sys.catalog]: This master's current role is: LEADER
I20260928 21:54:30.589196 9665 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8e56bb6247ad4b39a8bf4340451035a5 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 8e56bb6247ad4b39a8bf4340451035a5 (127.2.182.124) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d22a18a4a71545768146dab94c9a04a1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43551 } } peers { permanent_uuid: "59256c4c13d94676ac3e3b7e1e51a6a8" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 33661 } attrs { promote: false } } peers { permanent_uuid: "8e56bb6247ad4b39a8bf4340451035a5" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 40387 } attrs { promote: false } } }
I20260928 21:54:30.589870 10150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59256c4c13d94676ac3e3b7e1e51a6a8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d22a18a4a71545768146dab94c9a04a1" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d22a18a4a71545768146dab94c9a04a1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43551 } } peers { permanent_uuid: "59256c4c13d94676ac3e3b7e1e51a6a8" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 33661 } attrs { promote: false } } peers { permanent_uuid: "8e56bb6247ad4b39a8bf4340451035a5" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 40387 } attrs { promote: false } } }
I20260928 21:54:30.589946 10150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59256c4c13d94676ac3e3b7e1e51a6a8 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:54:30.590081 9694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e56bb6247ad4b39a8bf4340451035a5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d22a18a4a71545768146dab94c9a04a1" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d22a18a4a71545768146dab94c9a04a1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43551 } } peers { permanent_uuid: "59256c4c13d94676ac3e3b7e1e51a6a8" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 33661 } attrs { promote: false } } peers { permanent_uuid: "8e56bb6247ad4b39a8bf4340451035a5" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 40387 } attrs { promote: false } } }
I20260928 21:54:30.590148 9694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e56bb6247ad4b39a8bf4340451035a5 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 21:54:30.595839 9837 tablet.cc:2369] T e44118a0519846ec81d8f2d74750c6fb P 4438ab182bef4715bd7a150fab1403bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:54:30.597011 10131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:30.594587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49140 (local address 127.2.182.67:33249)
0928 21:54:30.594639 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:30.594641 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:30.594650 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:54:30.594688 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:30.594689 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:30.594709 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:30.594772 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:54:30.594967 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:30.595576 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:30.596340 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:30.596542 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:30.596594 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:30.596724 (+ 130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:54:30.596826 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:54:30.596897 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:30.596917 (+ 20us) server_negotiation.cc:299] Negotiation successful
0928 21:54:30.596938 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 21:54:30.598757 9788 raft_consensus.cc:1261] T e44118a0519846ec81d8f2d74750c6fb P 4438ab182bef4715bd7a150fab1403bd [term 1 FOLLOWER]: Refusing update from remote peer 65646a62d4364636adf7ec56574214c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:54:30.598918 9925 raft_consensus.cc:1261] T e44118a0519846ec81d8f2d74750c6fb P d6cf3fea18684e48bc758ab6bf07563e [term 1 FOLLOWER]: Refusing update from remote peer 65646a62d4364636adf7ec56574214c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:54:30.599148 10145 consensus_queue.cc:1035] T e44118a0519846ec81d8f2d74750c6fb P 65646a62d4364636adf7ec56574214c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4438ab182bef4715bd7a150fab1403bd" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 39731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:54:30.599282 10141 consensus_queue.cc:1035] T e44118a0519846ec81d8f2d74750c6fb P 65646a62d4364636adf7ec56574214c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6cf3fea18684e48bc758ab6bf07563e" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:54:30.601790 10153 mvcc.cc:205] Tried to move back new op lower bound from 7334430599569874944 to 7334430599514284032. Current Snapshot: MvccSnapshot[applied={T|T < 7334430599569874944}]
I20260928 21:54:30.601961 10152 mvcc.cc:205] Tried to move back new op lower bound from 7334430599569874944 to 7334430599514284032. Current Snapshot: MvccSnapshot[applied={T|T < 7334430599569874944}]
I20260928 21:54:30.602789 10154 mvcc.cc:205] Tried to move back new op lower bound from 7334430599569874944 to 7334430599514284032. Current Snapshot: MvccSnapshot[applied={T|T < 7334430599569874944}]
I20260928 21:54:30.606369 8869 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46408:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:54:30.606484 8869 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46408:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:54:30.608162 8869 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d22a18a4a71545768146dab94c9a04a1: Sending DeleteTablet for 3 replicas of tablet e44118a0519846ec81d8f2d74750c6fb
I20260928 21:54:30.608532 10041 tablet_service.cc:1514] Processing DeleteTablet for tablet e44118a0519846ec81d8f2d74750c6fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:54:30 UTC) from {username='oryx', principal='oryx/127.2.182.126@KRBTEST.COM'} at 127.0.0.1:49128
I20260928 21:54:30.608567 9905 tablet_service.cc:1514] Processing DeleteTablet for tablet e44118a0519846ec81d8f2d74750c6fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:54:30 UTC) from {username='oryx', principal='oryx/127.2.182.126@KRBTEST.COM'} at 127.0.0.1:34384
I20260928 21:54:30.608863 9768 tablet_service.cc:1514] Processing DeleteTablet for tablet e44118a0519846ec81d8f2d74750c6fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:54:30 UTC) from {username='oryx', principal='oryx/127.2.182.126@KRBTEST.COM'} at 127.0.0.1:56200
I20260928 21:54:30.608938 10161 tablet_replica.cc:330] T e44118a0519846ec81d8f2d74750c6fb P d6cf3fea18684e48bc758ab6bf07563e: stopping tablet replica
I20260928 21:54:30.609051 10161 raft_consensus.cc:2229] T e44118a0519846ec81d8f2d74750c6fb P d6cf3fea18684e48bc758ab6bf07563e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:54:30.609236 10161 raft_consensus.cc:2258] T e44118a0519846ec81d8f2d74750c6fb P d6cf3fea18684e48bc758ab6bf07563e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:54:30.609973 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 8838
I20260928 21:54:30.610051 10161 ts_tablet_manager.cc:1905] T e44118a0519846ec81d8f2d74750c6fb P d6cf3fea18684e48bc758ab6bf07563e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:54:30.612478 10161 ts_tablet_manager.cc:1918] T e44118a0519846ec81d8f2d74750c6fb P d6cf3fea18684e48bc758ab6bf07563e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:54:30.612552 10161 log.cc:1196] T e44118a0519846ec81d8f2d74750c6fb P d6cf3fea18684e48bc758ab6bf07563e: Deleting WAL directory at /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-1/wal/wals/e44118a0519846ec81d8f2d74750c6fb
I20260928 21:54:30.612814 10161 ts_tablet_manager.cc:1939] T e44118a0519846ec81d8f2d74750c6fb P d6cf3fea18684e48bc758ab6bf07563e: Deleting consensus metadata
I20260928 21:54:30.614177 10162 tablet_replica.cc:330] T e44118a0519846ec81d8f2d74750c6fb P 65646a62d4364636adf7ec56574214c0: stopping tablet replica
I20260928 21:54:30.614387 10162 raft_consensus.cc:2229] T e44118a0519846ec81d8f2d74750c6fb P 65646a62d4364636adf7ec56574214c0 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:54:30.614609 10162 raft_consensus.cc:2258] T e44118a0519846ec81d8f2d74750c6fb P 65646a62d4364636adf7ec56574214c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:54:30.614786 10163 tablet_replica.cc:330] T e44118a0519846ec81d8f2d74750c6fb P 4438ab182bef4715bd7a150fab1403bd: stopping tablet replica
I20260928 21:54:30.614954 10163 raft_consensus.cc:2229] T e44118a0519846ec81d8f2d74750c6fb P 4438ab182bef4715bd7a150fab1403bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:54:30.615203 10163 raft_consensus.cc:2258] T e44118a0519846ec81d8f2d74750c6fb P 4438ab182bef4715bd7a150fab1403bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:54:30.615739 10162 ts_tablet_manager.cc:1905] T e44118a0519846ec81d8f2d74750c6fb P 65646a62d4364636adf7ec56574214c0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 21:54:30.616029 9861 connection.cc:535] server connection from 127.0.0.1:34384 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 21:54:30.616588 10163 ts_tablet_manager.cc:1905] T e44118a0519846ec81d8f2d74750c6fb P 4438ab182bef4715bd7a150fab1403bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:54:30.619120 9840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:30.609474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55136 (local address 127.2.182.124:40387)
0928 21:54:30.609523 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:30.609525 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:30.611529 (+ 2004us) server_negotiation.cc:407] Connection header received
0928 21:54:30.611630 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:30.611631 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:30.611642 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:30.611672 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:30.612430 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:30.613403 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:30.614106 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:30.614366 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:30.614430 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:30.616150 (+ 1720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:30.616152 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:30.616152 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:30.616160 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:30.617713 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:30.618125 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:30.618126 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:30.618126 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:30.618207 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:30.618542 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:30.618543 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:30.618543 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:30.618677 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:30.618752 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:30.618948 (+ 196us) server_negotiation.cc:299] Negotiation successful
0928 21:54:30.619032 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 21:54:30.620223 10163 ts_tablet_manager.cc:1918] T e44118a0519846ec81d8f2d74750c6fb P 4438ab182bef4715bd7a150fab1403bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:54:30.620393 10162 ts_tablet_manager.cc:1918] T e44118a0519846ec81d8f2d74750c6fb P 65646a62d4364636adf7ec56574214c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:54:30.620515 10162 log.cc:1196] T e44118a0519846ec81d8f2d74750c6fb P 65646a62d4364636adf7ec56574214c0: Deleting WAL directory at /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-2/wal/wals/e44118a0519846ec81d8f2d74750c6fb
I20260928 21:54:30.620833 10162 ts_tablet_manager.cc:1939] T e44118a0519846ec81d8f2d74750c6fb P 65646a62d4364636adf7ec56574214c0: Deleting consensus metadata
I20260928 21:54:30.620951 10163 log.cc:1196] T e44118a0519846ec81d8f2d74750c6fb P 4438ab182bef4715bd7a150fab1403bd: Deleting WAL directory at /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-0/wal/wals/e44118a0519846ec81d8f2d74750c6fb
I20260928 21:54:30.621269 10163 ts_tablet_manager.cc:1939] T e44118a0519846ec81d8f2d74750c6fb P 4438ab182bef4715bd7a150fab1403bd: Deleting consensus metadata
W20260928 21:54:30.621654 9994 connection.cc:441] server connection from 127.0.0.1:49128 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49128 (request call id 1) could send its response
W20260928 21:54:30.622113 9723 connection.cc:441] server connection from 127.0.0.1:56200 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56200 (request call id 1) could send its response
I20260928 21:54:30.626384 10119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:30.618164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37076 (local address 127.2.182.125:33661)
0928 21:54:30.618234 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:30.618236 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:30.619096 (+ 860us) server_negotiation.cc:407] Connection header received
0928 21:54:30.619194 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:30.619195 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:30.619208 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:30.619236 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:30.620186 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:30.620890 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:30.621887 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:30.622177 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:30.622224 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:30.623709 (+ 1485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:30.623711 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:30.623711 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:30.623719 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:30.625230 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:30.625609 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:30.625610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:30.625610 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:30.625668 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:30.625856 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:30.625857 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:30.625857 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:30.625995 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:30.626093 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:30.626205 (+ 112us) server_negotiation.cc:299] Negotiation successful
0928 21:54:30.626291 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 21:54:30.627447 10121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:30.618674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55150 (local address 127.2.182.124:40387)
0928 21:54:30.618734 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:30.618736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:30.619501 (+ 765us) server_negotiation.cc:407] Connection header received
0928 21:54:30.619583 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:30.619584 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:30.619596 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:30.619622 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:30.620888 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:30.621655 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:30.622840 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:30.623137 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:30.623211 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:30.624864 (+ 1653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:30.624865 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:30.624865 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:30.624873 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:30.626310 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:30.626654 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:30.626655 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:30.626655 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:30.626715 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:30.626931 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:30.626932 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:30.626932 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:30.627075 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:30.627145 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:30.627271 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 21:54:30.627361 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 21:54:31.603497 10172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:31.603105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:43551 (local address 127.2.182.65:54963)
0928 21:54:31.603238 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:31.603406 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:43551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":65,"threads_started":1}
W20260928 21:54:31.603920 9835 heartbeater.cc:644] Failed to heartbeat to 127.2.182.126:43551 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.182.126:43551: connect: Connection refused (error 111)
I20260928 21:54:31.604117 10173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:31.603718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:43551 (local address 127.2.182.66:38455)
0928 21:54:31.604008 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:31.604076 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:43551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":43,"threads_started":1}
I20260928 21:54:31.604182 10174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:31.603883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:43551 (local address 127.2.182.67:44635)
0928 21:54:31.604088 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:31.604144 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:43551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":49,"threads_started":1}
W20260928 21:54:31.604270 9972 heartbeater.cc:644] Failed to heartbeat to 127.2.182.126:43551 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.182.126:43551: connect: Connection refused (error 111)
W20260928 21:54:31.604413 10108 heartbeater.cc:644] Failed to heartbeat to 127.2.182.126:43551 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.182.126:43551: connect: Connection refused (error 111)
I20260928 21:54:32.109040 10176 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e56bb6247ad4b39a8bf4340451035a5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d22a18a4a71545768146dab94c9a04a1)
I20260928 21:54:32.109086 10175 raft_consensus.cc:482] T 00000000000000000000000000000000 P 59256c4c13d94676ac3e3b7e1e51a6a8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d22a18a4a71545768146dab94c9a04a1)
I20260928 21:54:32.109166 10176 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e56bb6247ad4b39a8bf4340451035a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d22a18a4a71545768146dab94c9a04a1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43551 } } peers { permanent_uuid: "59256c4c13d94676ac3e3b7e1e51a6a8" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 33661 } attrs { promote: false } } peers { permanent_uuid: "8e56bb6247ad4b39a8bf4340451035a5" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 40387 } attrs { promote: false } }
I20260928 21:54:32.109206 10175 raft_consensus.cc:504] T 00000000000000000000000000000000 P 59256c4c13d94676ac3e3b7e1e51a6a8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d22a18a4a71545768146dab94c9a04a1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43551 } } peers { permanent_uuid: "59256c4c13d94676ac3e3b7e1e51a6a8" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 33661 } attrs { promote: false } } peers { permanent_uuid: "8e56bb6247ad4b39a8bf4340451035a5" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 40387 } attrs { promote: false } }
I20260928 21:54:32.109478 10176 leader_election.cc:288] T 00000000000000000000000000000000 P 8e56bb6247ad4b39a8bf4340451035a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d22a18a4a71545768146dab94c9a04a1 (127.2.182.126:43551), 59256c4c13d94676ac3e3b7e1e51a6a8 (127.2.182.125:33661)
I20260928 21:54:32.109434 10175 leader_election.cc:288] T 00000000000000000000000000000000 P 59256c4c13d94676ac3e3b7e1e51a6a8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d22a18a4a71545768146dab94c9a04a1 (127.2.182.126:43551), 8e56bb6247ad4b39a8bf4340451035a5 (127.2.182.124:40387)
I20260928 21:54:32.109782 10177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:32.109555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:43551 (local address 127.0.0.1:46864)
0928 21:54:32.109675 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:32.109738 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:43551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":53,"threads_started":1}
W20260928 21:54:32.109891 9452 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.182.126:43551: connect: Connection refused (error 111)
I20260928 21:54:32.109912 10181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:32.109566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:43551 (local address 127.0.0.1:46868)
0928 21:54:32.109841 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:32.109882 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:43551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":53,"threads_started":1}
W20260928 21:54:32.109987 9639 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.182.126:43551: connect: Connection refused (error 111)
I20260928 21:54:32.110296 10177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:32.110216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:43551 (local address 127.0.0.1:46878)
0928 21:54:32.110249 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:32.110271 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:43551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260928 21:54:32.110311 10181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:32.110245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:43551 (local address 127.0.0.1:46890)
0928 21:54:32.110274 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:32.110292 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:43551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260928 21:54:32.110385 9639 leader_election.cc:334] T 00000000000000000000000000000000 P 8e56bb6247ad4b39a8bf4340451035a5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d22a18a4a71545768146dab94c9a04a1 (127.2.182.126:43551): Network error: Client connection negotiation failed: client connection to 127.2.182.126:43551: connect: Connection refused (error 111)
W20260928 21:54:32.110383 9452 leader_election.cc:334] T 00000000000000000000000000000000 P 59256c4c13d94676ac3e3b7e1e51a6a8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d22a18a4a71545768146dab94c9a04a1 (127.2.182.126:43551): Network error: Client connection negotiation failed: client connection to 127.2.182.126:43551: connect: Connection refused (error 111)
I20260928 21:54:32.112939 10178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:32.109749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.125:33661 (local address 127.0.0.1:37078)
0928 21:54:32.109976 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:32.109986 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:32.110027 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:32.110214 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:32.110215 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:32.110217 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:54:32.110482 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:32.110484 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:32.111350 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:32.111351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:32.112166 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:32.112168 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:32.112717 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:32.112718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:32.112785 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:32.112792 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 21:54:32.112841 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 21:54:32.112881 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":37,"threads_started":1}
I20260928 21:54:32.113027 10180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:32.109685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37078 (local address 127.2.182.125:33661)
0928 21:54:32.110077 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:32.110079 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:32.110086 (+ 7us) server_negotiation.cc:407] Connection header received
0928 21:54:32.110109 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:32.110110 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:32.110120 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:32.110543 (+ 423us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:54:32.110574 (+ 31us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:32.111247 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:32.112252 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:32.112638 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:32.112702 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:32.112718 (+ 16us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:32.112908 (+ 190us) server_negotiation.cc:299] Negotiation successful
0928 21:54:32.112957 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":34,"threads_started":1}
I20260928 21:54:32.113140 10179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:32.109573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.124:40387 (local address 127.0.0.1:55154)
0928 21:54:32.109725 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:32.109736 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:32.109886 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:32.110216 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:32.110216 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:32.110218 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:54:32.110280 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:32.110281 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:32.111265 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:32.111266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:32.112155 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:32.112158 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:32.112917 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:32.112918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:32.112980 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:32.112984 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:54:32.113046 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 21:54:32.113086 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":45,"threads_started":1}
I20260928 21:54:32.113238 9478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8e56bb6247ad4b39a8bf4340451035a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "59256c4c13d94676ac3e3b7e1e51a6a8" is_pre_election: true
I20260928 21:54:32.113314 10182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:32.109760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55154 (local address 127.2.182.124:40387)
0928 21:54:32.110102 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:32.110103 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:32.110108 (+ 5us) server_negotiation.cc:407] Connection header received
0928 21:54:32.110124 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:32.110124 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:32.110132 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:32.110340 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:54:32.110368 (+ 28us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:32.111161 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:32.112248 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:32.112687 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:32.112785 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:32.112796 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:32.113212 (+ 416us) server_negotiation.cc:299] Negotiation successful
0928 21:54:32.113261 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":52,"threads_started":1}
I20260928 21:54:32.113350 9478 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 59256c4c13d94676ac3e3b7e1e51a6a8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e56bb6247ad4b39a8bf4340451035a5 in term 1.
I20260928 21:54:32.113512 9640 leader_election.cc:302] T 00000000000000000000000000000000 P 8e56bb6247ad4b39a8bf4340451035a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59256c4c13d94676ac3e3b7e1e51a6a8, 8e56bb6247ad4b39a8bf4340451035a5; no voters: d22a18a4a71545768146dab94c9a04a1
I20260928 21:54:32.113490 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "59256c4c13d94676ac3e3b7e1e51a6a8" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8e56bb6247ad4b39a8bf4340451035a5" is_pre_election: true
I20260928 21:54:32.113584 9665 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8e56bb6247ad4b39a8bf4340451035a5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59256c4c13d94676ac3e3b7e1e51a6a8 in term 1.
I20260928 21:54:32.113626 10176 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e56bb6247ad4b39a8bf4340451035a5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 21:54:32.113705 10176 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e56bb6247ad4b39a8bf4340451035a5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d22a18a4a71545768146dab94c9a04a1)
I20260928 21:54:32.113724 10176 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e56bb6247ad4b39a8bf4340451035a5 [term 1 FOLLOWER]: Advancing to term 2
I20260928 21:54:32.113821 9453 leader_election.cc:302] T 00000000000000000000000000000000 P 59256c4c13d94676ac3e3b7e1e51a6a8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59256c4c13d94676ac3e3b7e1e51a6a8, 8e56bb6247ad4b39a8bf4340451035a5; no voters: d22a18a4a71545768146dab94c9a04a1
I20260928 21:54:32.113978 10175 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 59256c4c13d94676ac3e3b7e1e51a6a8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 21:54:32.114030 10175 raft_consensus.cc:482] T 00000000000000000000000000000000 P 59256c4c13d94676ac3e3b7e1e51a6a8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d22a18a4a71545768146dab94c9a04a1)
I20260928 21:54:32.114051 10175 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59256c4c13d94676ac3e3b7e1e51a6a8 [term 1 FOLLOWER]: Advancing to term 2
I20260928 21:54:32.114473 10176 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e56bb6247ad4b39a8bf4340451035a5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d22a18a4a71545768146dab94c9a04a1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43551 } } peers { permanent_uuid: "59256c4c13d94676ac3e3b7e1e51a6a8" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 33661 } attrs { promote: false } } peers { permanent_uuid: "8e56bb6247ad4b39a8bf4340451035a5" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 40387 } attrs { promote: false } }
I20260928 21:54:32.114602 10176 leader_election.cc:288] T 00000000000000000000000000000000 P 8e56bb6247ad4b39a8bf4340451035a5 [CANDIDATE]: Term 2 election: Requested vote from peers d22a18a4a71545768146dab94c9a04a1 (127.2.182.126:43551), 59256c4c13d94676ac3e3b7e1e51a6a8 (127.2.182.125:33661)
I20260928 21:54:32.114773 9478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8e56bb6247ad4b39a8bf4340451035a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "59256c4c13d94676ac3e3b7e1e51a6a8"
I20260928 21:54:32.114835 10178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:32.114721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:43551 (local address 127.0.0.1:46904)
0928 21:54:32.114779 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:32.114806 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:43551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 21:54:32.114794 10175 raft_consensus.cc:504] T 00000000000000000000000000000000 P 59256c4c13d94676ac3e3b7e1e51a6a8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d22a18a4a71545768146dab94c9a04a1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43551 } } peers { permanent_uuid: "59256c4c13d94676ac3e3b7e1e51a6a8" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 33661 } attrs { promote: false } } peers { permanent_uuid: "8e56bb6247ad4b39a8bf4340451035a5" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 40387 } attrs { promote: false } }
I20260928 21:54:32.114934 9478 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 59256c4c13d94676ac3e3b7e1e51a6a8 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 8e56bb6247ad4b39a8bf4340451035a5 in current term 2: Already voted for candidate 59256c4c13d94676ac3e3b7e1e51a6a8 in this term.
I20260928 21:54:32.115365 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "59256c4c13d94676ac3e3b7e1e51a6a8" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8e56bb6247ad4b39a8bf4340451035a5"
I20260928 21:54:32.114939 10175 leader_election.cc:288] T 00000000000000000000000000000000 P 59256c4c13d94676ac3e3b7e1e51a6a8 [CANDIDATE]: Term 2 election: Requested vote from peers d22a18a4a71545768146dab94c9a04a1 (127.2.182.126:43551), 8e56bb6247ad4b39a8bf4340451035a5 (127.2.182.124:40387)
I20260928 21:54:32.115463 9665 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 8e56bb6247ad4b39a8bf4340451035a5 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 59256c4c13d94676ac3e3b7e1e51a6a8 in current term 2: Already voted for candidate 8e56bb6247ad4b39a8bf4340451035a5 in this term.
I20260928 21:54:32.115520 10179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:32.115022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:43551 (local address 127.0.0.1:46908)
0928 21:54:32.115461 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:32.115492 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:43551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422}
I20260928 21:54:32.115720 10178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:32.115376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:43551 (local address 127.0.0.1:46918)
0928 21:54:32.115677 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:32.115701 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:43551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":285}
W20260928 21:54:32.115784 9639 leader_election.cc:334] T 00000000000000000000000000000000 P 8e56bb6247ad4b39a8bf4340451035a5 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer d22a18a4a71545768146dab94c9a04a1 (127.2.182.126:43551): Network error: Client connection negotiation failed: client connection to 127.2.182.126:43551: connect: Connection refused (error 111)
I20260928 21:54:32.115824 9639 leader_election.cc:302] T 00000000000000000000000000000000 P 8e56bb6247ad4b39a8bf4340451035a5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e56bb6247ad4b39a8bf4340451035a5; no voters: 59256c4c13d94676ac3e3b7e1e51a6a8, d22a18a4a71545768146dab94c9a04a1
I20260928 21:54:32.115883 10176 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 8e56bb6247ad4b39a8bf4340451035a5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260928 21:54:32.115981 10179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:32.115736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:43551 (local address 127.0.0.1:46934)
0928 21:54:32.115916 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:32.115956 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:43551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163}
W20260928 21:54:32.116055 9452 leader_election.cc:334] T 00000000000000000000000000000000 P 59256c4c13d94676ac3e3b7e1e51a6a8 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer d22a18a4a71545768146dab94c9a04a1 (127.2.182.126:43551): Network error: Client connection negotiation failed: client connection to 127.2.182.126:43551: connect: Connection refused (error 111)
I20260928 21:54:32.116097 9452 leader_election.cc:302] T 00000000000000000000000000000000 P 59256c4c13d94676ac3e3b7e1e51a6a8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59256c4c13d94676ac3e3b7e1e51a6a8; no voters: 8e56bb6247ad4b39a8bf4340451035a5, d22a18a4a71545768146dab94c9a04a1
I20260928 21:54:32.116164 10175 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 59256c4c13d94676ac3e3b7e1e51a6a8 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260928 21:54:32.604583 10185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:32.604251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:43551 (local address 127.2.182.65:50169)
0928 21:54:32.604486 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:32.604527 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:43551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":79,"threads_started":1}
I20260928 21:54:32.604813 10186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:32.604580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:43551 (local address 127.2.182.66:51169)
0928 21:54:32.604737 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:32.604769 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:43551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":58,"threads_started":1}
I20260928 21:54:32.604872 10187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:32.604674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:43551 (local address 127.2.182.67:48735)
0928 21:54:32.604803 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:32.604835 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:43551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":54,"threads_started":1}
I20260928 21:54:33.605430 10189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:33.605127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:43551 (local address 127.2.182.66:35035)
0928 21:54:33.605332 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:33.605373 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:43551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":41,"threads_started":1}
I20260928 21:54:33.605584 10188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:33.605090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:43551 (local address 127.2.182.65:35739)
0928 21:54:33.605501 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:33.605539 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:43551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":63,"threads_started":1}
I20260928 21:54:33.605957 10190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:33.605397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:43551 (local address 127.2.182.67:41429)
0928 21:54:33.605705 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:33.605911 (+ 206us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:43551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":47,"threads_started":1}
I20260928 21:54:33.728034 10191 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e56bb6247ad4b39a8bf4340451035a5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:54:33.728132 10191 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e56bb6247ad4b39a8bf4340451035a5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d22a18a4a71545768146dab94c9a04a1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43551 } } peers { permanent_uuid: "59256c4c13d94676ac3e3b7e1e51a6a8" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 33661 } attrs { promote: false } } peers { permanent_uuid: "8e56bb6247ad4b39a8bf4340451035a5" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 40387 } attrs { promote: false } }
I20260928 21:54:33.728336 10191 leader_election.cc:288] T 00000000000000000000000000000000 P 8e56bb6247ad4b39a8bf4340451035a5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d22a18a4a71545768146dab94c9a04a1 (127.2.182.126:43551), 59256c4c13d94676ac3e3b7e1e51a6a8 (127.2.182.125:33661)
I20260928 21:54:33.728559 9478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8e56bb6247ad4b39a8bf4340451035a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "59256c4c13d94676ac3e3b7e1e51a6a8" is_pre_election: true
I20260928 21:54:33.728652 9478 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 59256c4c13d94676ac3e3b7e1e51a6a8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e56bb6247ad4b39a8bf4340451035a5 in term 2.
I20260928 21:54:33.728637 10192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:33.728458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:43551 (local address 127.0.0.1:47144)
0928 21:54:33.728552 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:33.728597 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:43551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77,"thread_start_us":45,"threads_started":1}
I20260928 21:54:33.728789 9640 leader_election.cc:302] T 00000000000000000000000000000000 P 8e56bb6247ad4b39a8bf4340451035a5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59256c4c13d94676ac3e3b7e1e51a6a8, 8e56bb6247ad4b39a8bf4340451035a5; no voters:
I20260928 21:54:33.728922 10191 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e56bb6247ad4b39a8bf4340451035a5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260928 21:54:33.728971 10191 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e56bb6247ad4b39a8bf4340451035a5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:54:33.728987 10191 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e56bb6247ad4b39a8bf4340451035a5 [term 2 FOLLOWER]: Advancing to term 3
I20260928 21:54:33.729112 10192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:33.729033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:43551 (local address 127.0.0.1:47154)
0928 21:54:33.729064 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:33.729088 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:43551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260928 21:54:33.729179 9639 leader_election.cc:334] T 00000000000000000000000000000000 P 8e56bb6247ad4b39a8bf4340451035a5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d22a18a4a71545768146dab94c9a04a1 (127.2.182.126:43551): Network error: Client connection negotiation failed: client connection to 127.2.182.126:43551: connect: Connection refused (error 111)
I20260928 21:54:33.729729 10191 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e56bb6247ad4b39a8bf4340451035a5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d22a18a4a71545768146dab94c9a04a1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43551 } } peers { permanent_uuid: "59256c4c13d94676ac3e3b7e1e51a6a8" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 33661 } attrs { promote: false } } peers { permanent_uuid: "8e56bb6247ad4b39a8bf4340451035a5" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 40387 } attrs { promote: false } }
I20260928 21:54:33.729915 10191 leader_election.cc:288] T 00000000000000000000000000000000 P 8e56bb6247ad4b39a8bf4340451035a5 [CANDIDATE]: Term 3 election: Requested vote from peers d22a18a4a71545768146dab94c9a04a1 (127.2.182.126:43551), 59256c4c13d94676ac3e3b7e1e51a6a8 (127.2.182.125:33661)
I20260928 21:54:33.729993 9478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8e56bb6247ad4b39a8bf4340451035a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "59256c4c13d94676ac3e3b7e1e51a6a8"
I20260928 21:54:33.730049 10192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:33.729959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:43551 (local address 127.0.0.1:47162)
0928 21:54:33.729995 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:33.730022 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:43551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260928 21:54:33.730063 9478 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59256c4c13d94676ac3e3b7e1e51a6a8 [term 2 FOLLOWER]: Advancing to term 3
I20260928 21:54:33.730371 10192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:33.730229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:43551 (local address 127.0.0.1:47170)
0928 21:54:33.730327 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:33.730353 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:43551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260928 21:54:33.730444 9639 leader_election.cc:334] T 00000000000000000000000000000000 P 8e56bb6247ad4b39a8bf4340451035a5 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer d22a18a4a71545768146dab94c9a04a1 (127.2.182.126:43551): Network error: Client connection negotiation failed: client connection to 127.2.182.126:43551: connect: Connection refused (error 111)
I20260928 21:54:33.730742 9478 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 59256c4c13d94676ac3e3b7e1e51a6a8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e56bb6247ad4b39a8bf4340451035a5 in term 3.
I20260928 21:54:33.730892 9640 leader_election.cc:302] T 00000000000000000000000000000000 P 8e56bb6247ad4b39a8bf4340451035a5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59256c4c13d94676ac3e3b7e1e51a6a8, 8e56bb6247ad4b39a8bf4340451035a5; no voters: d22a18a4a71545768146dab94c9a04a1
I20260928 21:54:33.730973 10191 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e56bb6247ad4b39a8bf4340451035a5 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 21:54:33.731062 10191 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e56bb6247ad4b39a8bf4340451035a5 [term 3 LEADER]: Becoming Leader. State: Replica: 8e56bb6247ad4b39a8bf4340451035a5, State: Running, Role: LEADER
I20260928 21:54:33.731119 10191 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e56bb6247ad4b39a8bf4340451035a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d22a18a4a71545768146dab94c9a04a1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43551 } } peers { permanent_uuid: "59256c4c13d94676ac3e3b7e1e51a6a8" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 33661 } attrs { promote: false } } peers { permanent_uuid: "8e56bb6247ad4b39a8bf4340451035a5" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 40387 } attrs { promote: false } }
I20260928 21:54:33.731508 10194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e56bb6247ad4b39a8bf4340451035a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e56bb6247ad4b39a8bf4340451035a5. Latest consensus state: current_term: 3 leader_uuid: "8e56bb6247ad4b39a8bf4340451035a5" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d22a18a4a71545768146dab94c9a04a1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43551 } } peers { permanent_uuid: "59256c4c13d94676ac3e3b7e1e51a6a8" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 33661 } attrs { promote: false } } peers { permanent_uuid: "8e56bb6247ad4b39a8bf4340451035a5" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 40387 } attrs { promote: false } } }
I20260928 21:54:33.731606 10194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e56bb6247ad4b39a8bf4340451035a5 [sys.catalog]: This master's current role is: LEADER
I20260928 21:54:33.731775 10196 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:54:33.732201 10196 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:54:33.732281 10196 catalog_manager.cc:1259] Loaded cluster ID: 088e5e211b3d41aa93ae71942e3a4aab
I20260928 21:54:33.732312 10196 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:54:33.732475 10196 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:54:33.732542 10196 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8e56bb6247ad4b39a8bf4340451035a5: Loaded TSK: 0
I20260928 21:54:33.732673 10196 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:54:33.791749 9655 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55150:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:54:33.793438 9655 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:54:33.794582 9478 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 59256c4c13d94676ac3e3b7e1e51a6a8 [term 3 FOLLOWER]: Refusing update from remote peer 8e56bb6247ad4b39a8bf4340451035a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260928 21:54:33.794610 10192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:33.794458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:43551 (local address 127.0.0.1:47198)
0928 21:54:33.794531 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:33.794574 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:43551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 21:54:33.794807 10191 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8e56bb6247ad4b39a8bf4340451035a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59256c4c13d94676ac3e3b7e1e51a6a8" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 33661 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260928 21:54:33.795043 10192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:33.794928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:43551 (local address 127.0.0.1:47202)
0928 21:54:33.794980 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:33.795012 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:43551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 21:54:33.795128 9639 consensus_peers.cc:487] T 00000000000000000000000000000000 P 8e56bb6247ad4b39a8bf4340451035a5 -> Peer d22a18a4a71545768146dab94c9a04a1 (127.2.182.126:43551): Couldn't send request to peer d22a18a4a71545768146dab94c9a04a1. Status: Network error: Client connection negotiation failed: client connection to 127.2.182.126:43551: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 21:54:33.795454 10198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59256c4c13d94676ac3e3b7e1e51a6a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e56bb6247ad4b39a8bf4340451035a5. Latest consensus state: current_term: 3 leader_uuid: "8e56bb6247ad4b39a8bf4340451035a5" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d22a18a4a71545768146dab94c9a04a1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43551 } } peers { permanent_uuid: "59256c4c13d94676ac3e3b7e1e51a6a8" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 33661 } attrs { promote: false } } peers { permanent_uuid: "8e56bb6247ad4b39a8bf4340451035a5" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 40387 } attrs { promote: false } } }
I20260928 21:54:33.795540 10198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59256c4c13d94676ac3e3b7e1e51a6a8 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:54:33.795938 10191 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e56bb6247ad4b39a8bf4340451035a5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "8e56bb6247ad4b39a8bf4340451035a5" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d22a18a4a71545768146dab94c9a04a1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43551 } } peers { permanent_uuid: "59256c4c13d94676ac3e3b7e1e51a6a8" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 33661 } attrs { promote: false } } peers { permanent_uuid: "8e56bb6247ad4b39a8bf4340451035a5" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 40387 } attrs { promote: false } } }
I20260928 21:54:33.796015 10191 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e56bb6247ad4b39a8bf4340451035a5 [sys.catalog]: This master's current role is: LEADER
I20260928 21:54:33.795994 10198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59256c4c13d94676ac3e3b7e1e51a6a8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "8e56bb6247ad4b39a8bf4340451035a5" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d22a18a4a71545768146dab94c9a04a1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43551 } } peers { permanent_uuid: "59256c4c13d94676ac3e3b7e1e51a6a8" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 33661 } attrs { promote: false } } peers { permanent_uuid: "8e56bb6247ad4b39a8bf4340451035a5" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 40387 } attrs { promote: false } } }
I20260928 21:54:33.796072 10198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59256c4c13d94676ac3e3b7e1e51a6a8 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:54:33.796236 10199 mvcc.cc:205] Tried to move back new op lower bound from 7334430612660252672 to 7334430612403130368. Current Snapshot: MvccSnapshot[applied={T|T < 7334430612660252672 or (T in {7334430612660252672})}]
I20260928 21:54:33.801075 10205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:33.798125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.66:33051 (local address 127.0.0.1:34400)
0928 21:54:33.798281 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:33.798289 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:33.798340 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:33.798615 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:33.798616 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:33.798617 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:54:33.798643 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:33.798644 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:33.799502 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:33.799503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:33.800231 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:33.800233 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:33.800841 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:33.800841 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:33.800903 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:33.800907 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:54:33.800954 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 21:54:33.800997 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":65,"threads_started":1}
I20260928 21:54:33.801087 10206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:33.798204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34400 (local address 127.2.182.66:33051)
0928 21:54:33.798293 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:33.798295 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:33.798330 (+ 35us) server_negotiation.cc:407] Connection header received
0928 21:54:33.798438 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:33.798439 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:33.798450 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:33.798617 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:54:33.798690 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:33.799438 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:33.800324 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:33.800770 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:33.800830 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:33.800843 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:33.800964 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 21:54:33.801000 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72,"thread_start_us":44,"threads_started":1}
I20260928 21:54:33.801483 10203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:33.797932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.67:33249 (local address 127.0.0.1:49152)
0928 21:54:33.798058 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:33.798068 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:33.798114 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:33.798509 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:33.798510 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:33.798511 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:54:33.798555 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:33.798557 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:33.799198 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:33.799198 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:33.800509 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:33.800511 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:33.801253 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:33.801254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:33.801312 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:33.801316 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:54:33.801370 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 21:54:33.801411 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":56,"thread_start_us":81,"threads_started":1}
I20260928 21:54:33.801497 10207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:33.798281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49152 (local address 127.2.182.67:33249)
0928 21:54:33.798378 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:33.798380 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:33.798387 (+ 7us) server_negotiation.cc:407] Connection header received
0928 21:54:33.798410 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:33.798410 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:33.798420 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:33.798508 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:54:33.798613 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:33.799121 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:33.800592 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:33.801145 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:33.801247 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:33.801262 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:33.801385 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 21:54:33.801431 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":51,"threads_started":1}
I20260928 21:54:33.801599 10204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:33.797971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56214 (local address 127.2.182.65:39731)
0928 21:54:33.798071 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:33.798073 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:33.798079 (+ 6us) server_negotiation.cc:407] Connection header received
0928 21:54:33.798103 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:33.798103 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:33.798113 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:33.798414 (+ 301us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:54:33.798453 (+ 39us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:33.799205 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:33.800563 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:33.801139 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:33.801454 (+ 315us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:33.801469 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:33.801486 (+ 17us) server_negotiation.cc:299] Negotiation successful
0928 21:54:33.801526 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":60,"threads_started":1}
I20260928 21:54:33.801635 10192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:33.797895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.65:39731 (local address 127.0.0.1:56214)
0928 21:54:33.797924 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:33.797934 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:33.797983 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:33.798198 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:33.798199 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:33.798201 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:54:33.798257 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:33.798259 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:33.799266 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:33.799267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:33.800463 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:33.800466 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:33.801236 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:33.801237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:33.801299 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:33.801303 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:54:33.801329 (+ 26us) client_negotiation.cc:240] Negotiation successful
0928 21:54:33.801369 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15,"mutex_wait_us":45}
I20260928 21:54:33.801663 9905 tablet_service.cc:1467] Processing CreateTablet for tablet 2cda492f197a43a59d49eb98b2b16fa8 (DEFAULT_TABLE table=test-table [id=219fd14ea34d4a7ca621995cf8e2a21a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:54:33.801783 9905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cda492f197a43a59d49eb98b2b16fa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:54:33.803081 10208 tablet_bootstrap.cc:492] T 2cda492f197a43a59d49eb98b2b16fa8 P d6cf3fea18684e48bc758ab6bf07563e: Bootstrap starting.
I20260928 21:54:33.803248 9768 tablet_service.cc:1467] Processing CreateTablet for tablet 2cda492f197a43a59d49eb98b2b16fa8 (DEFAULT_TABLE table=test-table [id=219fd14ea34d4a7ca621995cf8e2a21a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:54:33.803421 9768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cda492f197a43a59d49eb98b2b16fa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:54:33.803495 10208 tablet_bootstrap.cc:654] T 2cda492f197a43a59d49eb98b2b16fa8 P d6cf3fea18684e48bc758ab6bf07563e: Neither blocks nor log segments found. Creating new log.
I20260928 21:54:33.804656 10208 tablet_bootstrap.cc:492] T 2cda492f197a43a59d49eb98b2b16fa8 P d6cf3fea18684e48bc758ab6bf07563e: No bootstrap required, opened a new log
I20260928 21:54:33.804700 10208 ts_tablet_manager.cc:1397] T 2cda492f197a43a59d49eb98b2b16fa8 P d6cf3fea18684e48bc758ab6bf07563e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:54:33.804909 10208 raft_consensus.cc:348] T 2cda492f197a43a59d49eb98b2b16fa8 P d6cf3fea18684e48bc758ab6bf07563e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4438ab182bef4715bd7a150fab1403bd" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 39731 } } peers { permanent_uuid: "65646a62d4364636adf7ec56574214c0" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 33249 } } peers { permanent_uuid: "d6cf3fea18684e48bc758ab6bf07563e" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 33051 } }
I20260928 21:54:33.804903 10210 tablet_bootstrap.cc:492] T 2cda492f197a43a59d49eb98b2b16fa8 P 4438ab182bef4715bd7a150fab1403bd: Bootstrap starting.
I20260928 21:54:33.804950 10041 tablet_service.cc:1467] Processing CreateTablet for tablet 2cda492f197a43a59d49eb98b2b16fa8 (DEFAULT_TABLE table=test-table [id=219fd14ea34d4a7ca621995cf8e2a21a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:54:33.804982 10208 raft_consensus.cc:374] T 2cda492f197a43a59d49eb98b2b16fa8 P d6cf3fea18684e48bc758ab6bf07563e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:54:33.805001 10208 raft_consensus.cc:729] T 2cda492f197a43a59d49eb98b2b16fa8 P d6cf3fea18684e48bc758ab6bf07563e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6cf3fea18684e48bc758ab6bf07563e, State: Initialized, Role: FOLLOWER
I20260928 21:54:33.805058 10041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cda492f197a43a59d49eb98b2b16fa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:54:33.805061 10208 consensus_queue.cc:260] T 2cda492f197a43a59d49eb98b2b16fa8 P d6cf3fea18684e48bc758ab6bf07563e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4438ab182bef4715bd7a150fab1403bd" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 39731 } } peers { permanent_uuid: "65646a62d4364636adf7ec56574214c0" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 33249 } } peers { permanent_uuid: "d6cf3fea18684e48bc758ab6bf07563e" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 33051 } }
I20260928 21:54:33.805230 10208 ts_tablet_manager.cc:1428] T 2cda492f197a43a59d49eb98b2b16fa8 P d6cf3fea18684e48bc758ab6bf07563e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:33.805495 10210 tablet_bootstrap.cc:654] T 2cda492f197a43a59d49eb98b2b16fa8 P 4438ab182bef4715bd7a150fab1403bd: Neither blocks nor log segments found. Creating new log.
I20260928 21:54:33.806216 10210 tablet_bootstrap.cc:492] T 2cda492f197a43a59d49eb98b2b16fa8 P 4438ab182bef4715bd7a150fab1403bd: No bootstrap required, opened a new log
I20260928 21:54:33.806252 10210 ts_tablet_manager.cc:1397] T 2cda492f197a43a59d49eb98b2b16fa8 P 4438ab182bef4715bd7a150fab1403bd: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:54:33.806361 10213 tablet_bootstrap.cc:492] T 2cda492f197a43a59d49eb98b2b16fa8 P 65646a62d4364636adf7ec56574214c0: Bootstrap starting.
I20260928 21:54:33.806525 10210 raft_consensus.cc:348] T 2cda492f197a43a59d49eb98b2b16fa8 P 4438ab182bef4715bd7a150fab1403bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4438ab182bef4715bd7a150fab1403bd" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 39731 } } peers { permanent_uuid: "65646a62d4364636adf7ec56574214c0" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 33249 } } peers { permanent_uuid: "d6cf3fea18684e48bc758ab6bf07563e" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 33051 } }
I20260928 21:54:33.806591 10210 raft_consensus.cc:374] T 2cda492f197a43a59d49eb98b2b16fa8 P 4438ab182bef4715bd7a150fab1403bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:54:33.806607 10210 raft_consensus.cc:729] T 2cda492f197a43a59d49eb98b2b16fa8 P 4438ab182bef4715bd7a150fab1403bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4438ab182bef4715bd7a150fab1403bd, State: Initialized, Role: FOLLOWER
I20260928 21:54:33.806633 10210 consensus_queue.cc:260] T 2cda492f197a43a59d49eb98b2b16fa8 P 4438ab182bef4715bd7a150fab1403bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4438ab182bef4715bd7a150fab1403bd" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 39731 } } peers { permanent_uuid: "65646a62d4364636adf7ec56574214c0" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 33249 } } peers { permanent_uuid: "d6cf3fea18684e48bc758ab6bf07563e" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 33051 } }
I20260928 21:54:33.806640 10189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:33.806546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:43551 (local address 127.2.182.66:38231)
0928 21:54:33.806584 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:33.806611 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:43551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260928 21:54:33.806779 10210 ts_tablet_manager.cc:1428] T 2cda492f197a43a59d49eb98b2b16fa8 P 4438ab182bef4715bd7a150fab1403bd: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 21:54:33.807054 10213 tablet_bootstrap.cc:654] T 2cda492f197a43a59d49eb98b2b16fa8 P 65646a62d4364636adf7ec56574214c0: Neither blocks nor log segments found. Creating new log.
I20260928 21:54:33.807125 10188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:33.807013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:43551 (local address 127.2.182.65:58881)
0928 21:54:33.807067 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:33.807098 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:43551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 21:54:33.807530 9834 heartbeater.cc:503] Master 127.2.182.124:40387 was elected leader, sending a full tablet report...
I20260928 21:54:33.807554 9971 heartbeater.cc:503] Master 127.2.182.124:40387 was elected leader, sending a full tablet report...
I20260928 21:54:33.807747 10189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:33.807651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:43551 (local address 127.2.182.66:44107)
0928 21:54:33.807700 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:33.807723 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:43551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260928 21:54:33.807770 10188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:33.807695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:43551 (local address 127.2.182.65:45361)
0928 21:54:33.807729 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:33.807747 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:43551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260928 21:54:33.808156 10213 tablet_bootstrap.cc:492] T 2cda492f197a43a59d49eb98b2b16fa8 P 65646a62d4364636adf7ec56574214c0: No bootstrap required, opened a new log
I20260928 21:54:33.808221 10213 ts_tablet_manager.cc:1397] T 2cda492f197a43a59d49eb98b2b16fa8 P 65646a62d4364636adf7ec56574214c0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:54:33.808410 10213 raft_consensus.cc:348] T 2cda492f197a43a59d49eb98b2b16fa8 P 65646a62d4364636adf7ec56574214c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4438ab182bef4715bd7a150fab1403bd" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 39731 } } peers { permanent_uuid: "65646a62d4364636adf7ec56574214c0" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 33249 } } peers { permanent_uuid: "d6cf3fea18684e48bc758ab6bf07563e" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 33051 } }
I20260928 21:54:33.808502 10213 raft_consensus.cc:374] T 2cda492f197a43a59d49eb98b2b16fa8 P 65646a62d4364636adf7ec56574214c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:54:33.808532 10213 raft_consensus.cc:729] T 2cda492f197a43a59d49eb98b2b16fa8 P 65646a62d4364636adf7ec56574214c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65646a62d4364636adf7ec56574214c0, State: Initialized, Role: FOLLOWER
I20260928 21:54:33.808575 10213 consensus_queue.cc:260] T 2cda492f197a43a59d49eb98b2b16fa8 P 65646a62d4364636adf7ec56574214c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4438ab182bef4715bd7a150fab1403bd" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 39731 } } peers { permanent_uuid: "65646a62d4364636adf7ec56574214c0" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 33249 } } peers { permanent_uuid: "d6cf3fea18684e48bc758ab6bf07563e" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 33051 } }
I20260928 21:54:33.808735 10213 ts_tablet_manager.cc:1428] T 2cda492f197a43a59d49eb98b2b16fa8 P 65646a62d4364636adf7ec56574214c0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:33.809058 10190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:33.808971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:43551 (local address 127.2.182.67:59523)
0928 21:54:33.809005 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:33.809032 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:43551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260928 21:54:33.809167 10107 heartbeater.cc:503] Master 127.2.182.124:40387 was elected leader, sending a full tablet report...
W20260928 21:54:33.886010 10110 tablet.cc:2369] T 2cda492f197a43a59d49eb98b2b16fa8 P 65646a62d4364636adf7ec56574214c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:54:33.980727 10214 raft_consensus.cc:482] T 2cda492f197a43a59d49eb98b2b16fa8 P 4438ab182bef4715bd7a150fab1403bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:54:33.980840 10214 raft_consensus.cc:504] T 2cda492f197a43a59d49eb98b2b16fa8 P 4438ab182bef4715bd7a150fab1403bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4438ab182bef4715bd7a150fab1403bd" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 39731 } } peers { permanent_uuid: "65646a62d4364636adf7ec56574214c0" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 33249 } } peers { permanent_uuid: "d6cf3fea18684e48bc758ab6bf07563e" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 33051 } }
I20260928 21:54:33.981268 10214 leader_election.cc:288] T 2cda492f197a43a59d49eb98b2b16fa8 P 4438ab182bef4715bd7a150fab1403bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65646a62d4364636adf7ec56574214c0 (127.2.182.67:33249), d6cf3fea18684e48bc758ab6bf07563e (127.2.182.66:33051)
I20260928 21:54:33.985495 10206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:33.981701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.65:41393 (local address 127.2.182.66:33051)
0928 21:54:33.981765 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:33.981767 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:33.981803 (+ 36us) server_negotiation.cc:407] Connection header received
0928 21:54:33.981851 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:33.981852 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:33.981862 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:33.981915 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:54:33.982042 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:33.982833 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:33.984433 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:33.985068 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:33.985150 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:33.985161 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:33.985395 (+ 234us) server_negotiation.cc:299] Negotiation successful
0928 21:54:33.985436 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 21:54:33.985493 10207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:33.981475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.65:33693 (local address 127.2.182.67:33249)
0928 21:54:33.981533 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:33.981535 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:33.981628 (+ 93us) server_negotiation.cc:407] Connection header received
0928 21:54:33.981660 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:33.981662 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:33.981672 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:33.981880 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:54:33.982044 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:33.982850 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:33.984452 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:33.985068 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:33.985150 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:33.985161 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:33.985374 (+ 213us) server_negotiation.cc:299] Negotiation successful
0928 21:54:33.985422 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 21:54:33.985492 10188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:33.981462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.67:33249 (local address 127.2.182.65:33693)
0928 21:54:33.981507 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:33.981519 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:33.981581 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:33.981889 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:33.981890 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:33.981892 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:54:33.981973 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:33.981975 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:33.982972 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:33.982973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:33.984279 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:33.984283 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:33.985203 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:33.985205 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:33.985268 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:33.985274 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 21:54:33.985338 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 21:54:33.985388 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22,"mutex_wait_us":2}
I20260928 21:54:33.985502 10217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:33.981577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.66:33051 (local address 127.2.182.65:41393)
0928 21:54:33.981752 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:33.981760 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:33.981806 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:33.981927 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:33.981927 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:33.981928 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:54:33.981973 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:33.981975 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:33.982958 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:33.982959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:33.984290 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:33.984291 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:33.985178 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:33.985179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:33.985308 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:33.985312 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:54:33.985362 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 21:54:33.985400 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":63,"thread_start_us":108,"threads_started":1}
I20260928 21:54:33.985759 9925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cda492f197a43a59d49eb98b2b16fa8" candidate_uuid: "4438ab182bef4715bd7a150fab1403bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6cf3fea18684e48bc758ab6bf07563e" is_pre_election: true
I20260928 21:54:33.985836 10061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cda492f197a43a59d49eb98b2b16fa8" candidate_uuid: "4438ab182bef4715bd7a150fab1403bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65646a62d4364636adf7ec56574214c0" is_pre_election: true
I20260928 21:54:33.985889 9925 raft_consensus.cc:2454] T 2cda492f197a43a59d49eb98b2b16fa8 P d6cf3fea18684e48bc758ab6bf07563e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4438ab182bef4715bd7a150fab1403bd in term 0.
I20260928 21:54:33.985991 10061 raft_consensus.cc:2454] T 2cda492f197a43a59d49eb98b2b16fa8 P 65646a62d4364636adf7ec56574214c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4438ab182bef4715bd7a150fab1403bd in term 0.
I20260928 21:54:33.986136 9721 leader_election.cc:302] T 2cda492f197a43a59d49eb98b2b16fa8 P 4438ab182bef4715bd7a150fab1403bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4438ab182bef4715bd7a150fab1403bd, d6cf3fea18684e48bc758ab6bf07563e; no voters:
I20260928 21:54:33.986323 10214 raft_consensus.cc:2781] T 2cda492f197a43a59d49eb98b2b16fa8 P 4438ab182bef4715bd7a150fab1403bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:54:33.986367 10214 raft_consensus.cc:482] T 2cda492f197a43a59d49eb98b2b16fa8 P 4438ab182bef4715bd7a150fab1403bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:54:33.986387 10214 raft_consensus.cc:3037] T 2cda492f197a43a59d49eb98b2b16fa8 P 4438ab182bef4715bd7a150fab1403bd [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:54:33.987100 10214 raft_consensus.cc:504] T 2cda492f197a43a59d49eb98b2b16fa8 P 4438ab182bef4715bd7a150fab1403bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4438ab182bef4715bd7a150fab1403bd" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 39731 } } peers { permanent_uuid: "65646a62d4364636adf7ec56574214c0" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 33249 } } peers { permanent_uuid: "d6cf3fea18684e48bc758ab6bf07563e" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 33051 } }
I20260928 21:54:33.987226 10214 leader_election.cc:288] T 2cda492f197a43a59d49eb98b2b16fa8 P 4438ab182bef4715bd7a150fab1403bd [CANDIDATE]: Term 1 election: Requested vote from peers 65646a62d4364636adf7ec56574214c0 (127.2.182.67:33249), d6cf3fea18684e48bc758ab6bf07563e (127.2.182.66:33051)
I20260928 21:54:33.987463 10061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cda492f197a43a59d49eb98b2b16fa8" candidate_uuid: "4438ab182bef4715bd7a150fab1403bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65646a62d4364636adf7ec56574214c0"
I20260928 21:54:33.987493 9925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cda492f197a43a59d49eb98b2b16fa8" candidate_uuid: "4438ab182bef4715bd7a150fab1403bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6cf3fea18684e48bc758ab6bf07563e"
I20260928 21:54:33.987543 10061 raft_consensus.cc:3037] T 2cda492f197a43a59d49eb98b2b16fa8 P 65646a62d4364636adf7ec56574214c0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:54:33.987560 9925 raft_consensus.cc:3037] T 2cda492f197a43a59d49eb98b2b16fa8 P d6cf3fea18684e48bc758ab6bf07563e [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:54:33.988219 10061 raft_consensus.cc:2454] T 2cda492f197a43a59d49eb98b2b16fa8 P 65646a62d4364636adf7ec56574214c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4438ab182bef4715bd7a150fab1403bd in term 1.
I20260928 21:54:33.988231 9925 raft_consensus.cc:2454] T 2cda492f197a43a59d49eb98b2b16fa8 P d6cf3fea18684e48bc758ab6bf07563e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4438ab182bef4715bd7a150fab1403bd in term 1.
I20260928 21:54:33.988401 9721 leader_election.cc:302] T 2cda492f197a43a59d49eb98b2b16fa8 P 4438ab182bef4715bd7a150fab1403bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4438ab182bef4715bd7a150fab1403bd, d6cf3fea18684e48bc758ab6bf07563e; no voters:
I20260928 21:54:33.988514 10214 raft_consensus.cc:2781] T 2cda492f197a43a59d49eb98b2b16fa8 P 4438ab182bef4715bd7a150fab1403bd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:54:33.988678 10214 raft_consensus.cc:686] T 2cda492f197a43a59d49eb98b2b16fa8 P 4438ab182bef4715bd7a150fab1403bd [term 1 LEADER]: Becoming Leader. State: Replica: 4438ab182bef4715bd7a150fab1403bd, State: Running, Role: LEADER
I20260928 21:54:33.988744 10214 consensus_queue.cc:237] T 2cda492f197a43a59d49eb98b2b16fa8 P 4438ab182bef4715bd7a150fab1403bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4438ab182bef4715bd7a150fab1403bd" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 39731 } } peers { permanent_uuid: "65646a62d4364636adf7ec56574214c0" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 33249 } } peers { permanent_uuid: "d6cf3fea18684e48bc758ab6bf07563e" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 33051 } }
I20260928 21:54:33.989113 10217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:33.989001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:43551 (local address 127.2.182.65:44313)
0928 21:54:33.989035 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:33.989081 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:43551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260928 21:54:33.989532 9654 catalog_manager.cc:5495] T 2cda492f197a43a59d49eb98b2b16fa8 P 4438ab182bef4715bd7a150fab1403bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 4438ab182bef4715bd7a150fab1403bd (127.2.182.65). New cstate: current_term: 1 leader_uuid: "4438ab182bef4715bd7a150fab1403bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4438ab182bef4715bd7a150fab1403bd" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 39731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65646a62d4364636adf7ec56574214c0" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 33249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6cf3fea18684e48bc758ab6bf07563e" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 33051 } health_report { overall_health: UNKNOWN } } }
I20260928 21:54:34.031396 10204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:34.028832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56220 (local address 127.2.182.65:39731)
0928 21:54:34.028889 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:34.028891 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:34.028897 (+ 6us) server_negotiation.cc:407] Connection header received
0928 21:54:34.028935 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:34.028936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:34.028960 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:34.029030 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:54:34.029169 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:34.029660 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:34.030521 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:34.030716 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:34.030789 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:34.031002 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:54:34.031114 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:54:34.031182 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:34.031253 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 21:54:34.031280 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 21:54:34.032552 9925 raft_consensus.cc:1261] T 2cda492f197a43a59d49eb98b2b16fa8 P d6cf3fea18684e48bc758ab6bf07563e [term 1 FOLLOWER]: Refusing update from remote peer 4438ab182bef4715bd7a150fab1403bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:54:34.032660 10061 raft_consensus.cc:1261] T 2cda492f197a43a59d49eb98b2b16fa8 P 65646a62d4364636adf7ec56574214c0 [term 1 FOLLOWER]: Refusing update from remote peer 4438ab182bef4715bd7a150fab1403bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:54:34.032796 10214 consensus_queue.cc:1035] T 2cda492f197a43a59d49eb98b2b16fa8 P 4438ab182bef4715bd7a150fab1403bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6cf3fea18684e48bc758ab6bf07563e" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:54:34.033066 10214 consensus_queue.cc:1035] T 2cda492f197a43a59d49eb98b2b16fa8 P 4438ab182bef4715bd7a150fab1403bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "65646a62d4364636adf7ec56574214c0" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 33249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:54:34.033519 10189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:34.033406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:43551 (local address 127.2.182.66:52365)
0928 21:54:34.033452 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:34.033486 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:43551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 21:54:34.034210 10190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:34.034100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:43551 (local address 127.2.182.67:38867)
0928 21:54:34.034139 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:34.034177 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:43551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260928 21:54:34.034976 10189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:34.034860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:43551 (local address 127.2.182.66:35267)
0928 21:54:34.034922 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:34.034946 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:43551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 21:54:34.035727 10217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:34.035615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:43551 (local address 127.2.182.65:54735)
0928 21:54:34.035646 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:34.035678 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:43551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260928 21:54:34.036134 10217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:34.036055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:43551 (local address 127.2.182.65:58373)
0928 21:54:34.036084 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:34.036108 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:43551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
I20260928 21:54:34.036801 10190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:34.036696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:43551 (local address 127.2.182.67:60405)
0928 21:54:34.036737 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:34.036771 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:43551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 21:54:34.037984 9654 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55150:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:54:34.038079 9654 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55150:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:54:34.039476 9654 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8e56bb6247ad4b39a8bf4340451035a5: Sending DeleteTablet for 3 replicas of tablet 2cda492f197a43a59d49eb98b2b16fa8
I20260928 21:54:34.039757 10041 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cda492f197a43a59d49eb98b2b16fa8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:54:34 UTC) from {username='oryx', principal='oryx/127.2.182.124@KRBTEST.COM'} at 127.0.0.1:49152
I20260928 21:54:34.039810 9905 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cda492f197a43a59d49eb98b2b16fa8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:54:34 UTC) from {username='oryx', principal='oryx/127.2.182.124@KRBTEST.COM'} at 127.0.0.1:34400
I20260928 21:54:34.039901 9768 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cda492f197a43a59d49eb98b2b16fa8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:54:34 UTC) from {username='oryx', principal='oryx/127.2.182.124@KRBTEST.COM'} at 127.0.0.1:56214
I20260928 21:54:34.040026 10232 tablet_replica.cc:330] T 2cda492f197a43a59d49eb98b2b16fa8 P d6cf3fea18684e48bc758ab6bf07563e: stopping tablet replica
I20260928 21:54:34.040083 10232 raft_consensus.cc:2229] T 2cda492f197a43a59d49eb98b2b16fa8 P d6cf3fea18684e48bc758ab6bf07563e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:54:34.040139 10232 raft_consensus.cc:2258] T 2cda492f197a43a59d49eb98b2b16fa8 P d6cf3fea18684e48bc758ab6bf07563e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:54:34.040254 10231 tablet_replica.cc:330] T 2cda492f197a43a59d49eb98b2b16fa8 P 65646a62d4364636adf7ec56574214c0: stopping tablet replica
I20260928 21:54:34.040380 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 9615
I20260928 21:54:34.040321 10231 raft_consensus.cc:2229] T 2cda492f197a43a59d49eb98b2b16fa8 P 65646a62d4364636adf7ec56574214c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:54:34.040993 10231 raft_consensus.cc:2258] T 2cda492f197a43a59d49eb98b2b16fa8 P 65646a62d4364636adf7ec56574214c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:54:34.041316 10231 ts_tablet_manager.cc:1905] T 2cda492f197a43a59d49eb98b2b16fa8 P 65646a62d4364636adf7ec56574214c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:54:34.042929 10231 ts_tablet_manager.cc:1918] T 2cda492f197a43a59d49eb98b2b16fa8 P 65646a62d4364636adf7ec56574214c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:54:34.042999 10231 log.cc:1196] T 2cda492f197a43a59d49eb98b2b16fa8 P 65646a62d4364636adf7ec56574214c0: Deleting WAL directory at /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632440055392-2777-0/minicluster-data/ts-2/wal/wals/2cda492f197a43a59d49eb98b2b16fa8
I20260928 21:54:34.043210 10231 ts_tablet_manager.cc:1939] T 2cda492f197a43a59d49eb98b2b16fa8 P 65646a62d4364636adf7ec56574214c0: Deleting consensus metadata
I20260928 21:54:34.043556 9639 catalog_manager.cc:4841] TS 65646a62d4364636adf7ec56574214c0 (127.2.182.67:33249): tablet 2cda492f197a43a59d49eb98b2b16fa8 (table test-table [id=219fd14ea34d4a7ca621995cf8e2a21a]) successfully deleted
I20260928 21:54:34.046463 10232 ts_tablet_manager.cc:1905] T 2cda492f197a43a59d49eb98b2b16fa8 P d6cf3fea18684e48bc758ab6bf07563e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 21:54:34.046502 10205 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:54:34.046340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.65:39731 (local address 127.0.0.1:56226)
0928 21:54:34.046401 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:34.046439 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.65:39731: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 21:54:34.046654 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 9842
W20260928 21:54:34.046664 9640 catalog_manager.cc:4584] TS 4438ab182bef4715bd7a150fab1403bd (127.2.182.65:39731): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2cda492f197a43a59d49eb98b2b16fa8: Network error: Client connection negotiation failed: client connection to 127.2.182.65:39731: connect: Connection reset by peer (error 104)
I20260928 21:54:34.051700 10205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:34.051560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.66:33051 (local address 127.0.0.1:34408)
0928 21:54:34.051625 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:34.051658 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.66:33051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 21:54:34.051692 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 9978
I20260928 21:54:34.057034 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 9334
I20260928 21:54:34.061537 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 9522
2026-09-28T21:54:34Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6478 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:54:34 dist-test-slave-k6lz krb5kdc[10236](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:54:34 dist-test-slave-k6lz krb5kdc[10236](info): set up 2 sockets
Sep 28 21:54:34 dist-test-slave-k6lz krb5kdc[10236](info): commencing operation
krb5kdc: starting...
W20260928 21:54:35.031781 2777 mini_kdc.cc:121] Time spent starting KDC: real 0.949s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Sep 28 21:54:35 dist-test-slave-k6lz krb5kdc[10236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:54:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:54:35Z Disabled control of system clock
WARNING: no policy specified for oryx/127.2.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/krb5kdc/oryx_127.2.182.126.keytab.
Entry for principal oryx/127.2.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/krb5kdc/oryx_127.2.182.126.keytab.
I20260928 21:54:35.160703 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:45217
--webserver_interface=127.2.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/krb5kdc/oryx_127.2.182.126.keytab
--principal=oryx/127.2.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:36925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.126:45217
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:54:35.239928 10249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:54:35.240140 10249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:54:35.240161 10249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:54:35.241297 10249 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:54:35.241333 10249 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 21:54:35.241353 10249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:54:35.241364 10249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:54:35.241374 10249 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:54:35.241384 10249 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:54:35.242873 10249 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:36925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.126:45217
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/krb5kdc/oryx_127.2.182.126.keytab
--principal=oryx/127.2.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:45217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10249
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:54:35.243072 10249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:54:35.243245 10249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:54:35.248021 10255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:35.248001 10254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:35.248001 10257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:35.248054 10249 server_base.cc:1034] running on GCE node
I20260928 21:54:35.248461 10249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:35.248731 10249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:35.249888 10249 hybrid_clock.cc:648] HybridClock initialized: now 1790632475249872 us; error 27 us; skew 500 ppm
Sep 28 21:54:35 dist-test-slave-k6lz krb5kdc[10236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632475, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:54:35.252286 10249 init.cc:377] Logged in from keytab as oryx/127.2.182.126@KRBTEST.COM (short username oryx)
I20260928 21:54:35.252681 10249 webserver.cc:460] Webserver started at http://127.2.182.126:43189/ using document root <none> and password file <none>
I20260928 21:54:35.252965 10249 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:35.253015 10249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:35.253131 10249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:54:35.254189 10249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/minicluster-data/master-0/data/instance:
uuid: "85cac027afc849588f7eae9726ee1eb1"
format_stamp: "Formatted at 2026-09-28 21:54:35 on dist-test-slave-k6lz"
server_key: "f0ddbd2262b4a7ba3a8a997ac4d24e9c"
server_key_iv: "b945b32133b5070385a925e71607ca5d"
server_key_version: "encryptionkey@0"
I20260928 21:54:35.254500 10249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/minicluster-data/master-0/wal/instance:
uuid: "85cac027afc849588f7eae9726ee1eb1"
format_stamp: "Formatted at 2026-09-28 21:54:35 on dist-test-slave-k6lz"
server_key: "f0ddbd2262b4a7ba3a8a997ac4d24e9c"
server_key_iv: "b945b32133b5070385a925e71607ca5d"
server_key_version: "encryptionkey@0"
I20260928 21:54:35.255872 10249 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 21:54:35.256660 10264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:35.256953 10249 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:54:35.257062 10249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/minicluster-data/master-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/minicluster-data/master-0/wal
uuid: "85cac027afc849588f7eae9726ee1eb1"
format_stamp: "Formatted at 2026-09-28 21:54:35 on dist-test-slave-k6lz"
server_key: "f0ddbd2262b4a7ba3a8a997ac4d24e9c"
server_key_iv: "b945b32133b5070385a925e71607ca5d"
server_key_version: "encryptionkey@0"
I20260928 21:54:35.257133 10249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:35.286335 10249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:54:35.288600 10249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:54:35.288757 10249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:54:35.293033 10249 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.126:45217
I20260928 21:54:35.293104 10326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.126:45217 every 8 connection(s)
I20260928 21:54:35.293409 10249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
I20260928 21:54:35.294039 10327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:54:35.295857 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 10249
I20260928 21:54:35.295956 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/minicluster-data/master-0/wal/instance
I20260928 21:54:35.296147 2777 external_mini_cluster.cc:1428] Setting key daf79708489e8d9010a0b350eef864b6
I20260928 21:54:35.296517 10327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85cac027afc849588f7eae9726ee1eb1: Bootstrap starting.
I20260928 21:54:35.297119 10327 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85cac027afc849588f7eae9726ee1eb1: Neither blocks nor log segments found. Creating new log.
I20260928 21:54:35.297384 10327 log.cc:824] T 00000000000000000000000000000000 P 85cac027afc849588f7eae9726ee1eb1: Log is configured to *not* fsync() on all Append() calls
I20260928 21:54:35.297967 10327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85cac027afc849588f7eae9726ee1eb1: No bootstrap required, opened a new log
I20260928 21:54:35.299758 10327 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85cac027afc849588f7eae9726ee1eb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85cac027afc849588f7eae9726ee1eb1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 45217 } }
I20260928 21:54:35.299901 10327 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85cac027afc849588f7eae9726ee1eb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:54:35.299932 10327 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85cac027afc849588f7eae9726ee1eb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85cac027afc849588f7eae9726ee1eb1, State: Initialized, Role: FOLLOWER
I20260928 21:54:35.300055 10327 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85cac027afc849588f7eae9726ee1eb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85cac027afc849588f7eae9726ee1eb1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 45217 } }
I20260928 21:54:35.300122 10327 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85cac027afc849588f7eae9726ee1eb1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:54:35.300153 10327 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85cac027afc849588f7eae9726ee1eb1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:54:35.300197 10327 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85cac027afc849588f7eae9726ee1eb1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:54:35.300822 10327 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85cac027afc849588f7eae9726ee1eb1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85cac027afc849588f7eae9726ee1eb1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 45217 } }
I20260928 21:54:35.300928 10327 leader_election.cc:302] T 00000000000000000000000000000000 P 85cac027afc849588f7eae9726ee1eb1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85cac027afc849588f7eae9726ee1eb1; no voters:
I20260928 21:54:35.301080 10327 leader_election.cc:288] T 00000000000000000000000000000000 P 85cac027afc849588f7eae9726ee1eb1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:54:35.301121 10332 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85cac027afc849588f7eae9726ee1eb1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:54:35.301287 10332 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85cac027afc849588f7eae9726ee1eb1 [term 1 LEADER]: Becoming Leader. State: Replica: 85cac027afc849588f7eae9726ee1eb1, State: Running, Role: LEADER
I20260928 21:54:35.301331 10327 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85cac027afc849588f7eae9726ee1eb1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:54:35.301373 10332 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85cac027afc849588f7eae9726ee1eb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85cac027afc849588f7eae9726ee1eb1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 45217 } }
I20260928 21:54:35.301851 10334 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85cac027afc849588f7eae9726ee1eb1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85cac027afc849588f7eae9726ee1eb1. Latest consensus state: current_term: 1 leader_uuid: "85cac027afc849588f7eae9726ee1eb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85cac027afc849588f7eae9726ee1eb1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 45217 } } }
I20260928 21:54:35.301851 10333 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85cac027afc849588f7eae9726ee1eb1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85cac027afc849588f7eae9726ee1eb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85cac027afc849588f7eae9726ee1eb1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 45217 } } }
I20260928 21:54:35.302014 10334 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85cac027afc849588f7eae9726ee1eb1 [sys.catalog]: This master's current role is: LEADER
I20260928 21:54:35.302024 10333 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85cac027afc849588f7eae9726ee1eb1 [sys.catalog]: This master's current role is: LEADER
I20260928 21:54:35.302243 10341 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 28 21:54:35 dist-test-slave-k6lz krb5kdc[10236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM
I20260928 21:54:35.302802 10341 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:54:35.304679 10341 catalog_manager.cc:1347] Generated new cluster ID: 2a89be24a66e4e87b31f2df9d3b24120
I20260928 21:54:35.304728 10341 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:54:35.306268 10330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:35.296662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43102 (local address 127.2.182.126:45217)
0928 21:54:35.296809 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:35.296816 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:35.296841 (+ 25us) server_negotiation.cc:407] Connection header received
0928 21:54:35.297293 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:35.297300 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:35.297348 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:35.297477 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:35.298057 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:35.298977 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:35.299738 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:35.300183 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:35.300294 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:35.303030 (+ 2736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:35.303034 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:35.303034 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:35.303045 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:35.304755 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:35.305127 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:35.305130 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:35.305130 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:35.305227 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:35.305418 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:35.305418 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:35.305419 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:35.305687 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:35.305822 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:35.305945 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 21:54:35.306070 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":61,"threads_started":1}
I20260928 21:54:35.322839 10341 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:54:35.323437 10341 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:54:35.330359 10341 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85cac027afc849588f7eae9726ee1eb1: Generated new TSK 0
I20260928 21:54:35.330546 10341 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.2.182.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.182.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.182.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/krb5kdc/oryx_127.2.182.65.keytab.
Entry for principal oryx/127.2.182.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/krb5kdc/oryx_127.2.182.65.keytab.
I20260928 21:54:35.359833 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.65:0
--local_ip_for_outbound_sockets=127.2.182.65
--webserver_interface=127.2.182.65
--webserver_port=0
--tserver_master_addrs=127.2.182.126:45217
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/krb5kdc/oryx_127.2.182.65.keytab
--principal=oryx/127.2.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:36925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:54:35.441250 10353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:54:35.441466 10353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:54:35.441497 10353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:54:35.442636 10353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:54:35.442677 10353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:54:35.442705 10353 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 21:54:35.442729 10353 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.65
I20260928 21:54:35.444249 10353 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:36925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/krb5kdc/oryx_127.2.182.65.keytab
--principal=oryx/127.2.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.182.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.182.126:45217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.65
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:54:35.444504 10353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:54:35.444701 10353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:54:35.449458 10359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:35.449496 10361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:35.449646 10358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:35.449631 10353 server_base.cc:1034] running on GCE node
I20260928 21:54:35.449939 10353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:35.450165 10353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:35.451324 10353 hybrid_clock.cc:648] HybridClock initialized: now 1790632475451309 us; error 26 us; skew 500 ppm
Sep 28 21:54:35 dist-test-slave-k6lz krb5kdc[10236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632475, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:54:35.453745 10353 init.cc:377] Logged in from keytab as oryx/127.2.182.65@KRBTEST.COM (short username oryx)
I20260928 21:54:35.454326 10353 webserver.cc:460] Webserver started at http://127.2.182.65:36999/ using document root <none> and password file <none>
I20260928 21:54:35.454546 10353 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:35.454595 10353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:35.454717 10353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:54:35.455789 10353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/minicluster-data/ts-0/data/instance:
uuid: "8c5ce1c5850d406fbd3f4eeca6fde124"
format_stamp: "Formatted at 2026-09-28 21:54:35 on dist-test-slave-k6lz"
server_key: "91937ae70c5aab588eb450e2a497e772"
server_key_iv: "9970323d4de47e8fc3678dc287005864"
server_key_version: "encryptionkey@0"
I20260928 21:54:35.456110 10353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/minicluster-data/ts-0/wal/instance:
uuid: "8c5ce1c5850d406fbd3f4eeca6fde124"
format_stamp: "Formatted at 2026-09-28 21:54:35 on dist-test-slave-k6lz"
server_key: "91937ae70c5aab588eb450e2a497e772"
server_key_iv: "9970323d4de47e8fc3678dc287005864"
server_key_version: "encryptionkey@0"
I20260928 21:54:35.457326 10353 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:54:35.457986 10368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:35.458132 10353 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:54:35.458209 10353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/minicluster-data/ts-0/wal
uuid: "8c5ce1c5850d406fbd3f4eeca6fde124"
format_stamp: "Formatted at 2026-09-28 21:54:35 on dist-test-slave-k6lz"
server_key: "91937ae70c5aab588eb450e2a497e772"
server_key_iv: "9970323d4de47e8fc3678dc287005864"
server_key_version: "encryptionkey@0"
I20260928 21:54:35.458268 10353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:35.494074 10353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:54:35.496371 10353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:54:35.496485 10353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:54:35.497005 10353 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:54:35.497045 10353 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:35.497068 10353 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:54:35.497080 10353 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:35.503208 10353 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.65:35485
I20260928 21:54:35.503289 10483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.65:35485 every 8 connection(s)
I20260928 21:54:35.503589 10353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
Sep 28 21:54:35 dist-test-slave-k6lz krb5kdc[10236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632475, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.65@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM
I20260928 21:54:35.505414 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 10353
I20260928 21:54:35.505517 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/minicluster-data/ts-0/wal/instance
I20260928 21:54:35.505704 2777 external_mini_cluster.cc:1428] Setting key bbb950cd26708172a49e7ac88ebdcd58
I20260928 21:54:35.507121 10330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:35.497502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.65:52725 (local address 127.2.182.126:45217)
0928 21:54:35.497568 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:35.497570 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:35.497785 (+ 215us) server_negotiation.cc:407] Connection header received
0928 21:54:35.498313 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:35.498314 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:35.498327 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:35.498407 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:35.499239 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:35.500026 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:35.501459 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:35.501748 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:35.501840 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:35.504356 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:35.504358 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:35.504358 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:35.504367 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:35.505633 (+ 1266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:35.506076 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:35.506077 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:35.506077 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:35.506138 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:35.506421 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:35.506421 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:35.506422 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:35.506623 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:35.506691 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:35.506937 (+ 246us) server_negotiation.cc:299] Negotiation successful
0928 21:54:35.507033 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 21:54:35.507216 10377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:35.497491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:45217 (local address 127.2.182.65:52725)
0928 21:54:35.497649 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:35.497690 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:35.498206 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:35.498487 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:35.498491 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:35.498833 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:35.499126 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:35.499133 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:35.500117 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:35.500118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:35.501381 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:35.501385 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:35.501849 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:35.501850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:35.502015 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:35.502629 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:35.502639 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:35.504241 (+ 1602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:35.505711 (+ 1470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:35.505719 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:35.505721 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:35.506007 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:35.506193 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:35.506194 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:35.506194 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:35.506329 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:35.506690 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:35.506696 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:35.506816 (+ 120us) client_negotiation.cc:769] Sending connection context
0928 21:54:35.506911 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 21:54:35.507042 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":84,"threads_started":1}
I20260928 21:54:35.508109 10484 heartbeater.cc:347] Connected to a master server at 127.2.182.126:45217
I20260928 21:54:35.508204 10484 heartbeater.cc:464] Registering TS with master...
I20260928 21:54:35.508425 10484 heartbeater.cc:511] Master 127.2.182.126:45217 requested a full tablet report, sending...
I20260928 21:54:35.508934 10280 ts_manager.cc:194] Registered new tserver with Master: 8c5ce1c5850d406fbd3f4eeca6fde124 (127.2.182.65:35485)
I20260928 21:54:35.509585 10280 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.182.65@KRBTEST.COM'} at 127.2.182.65:52725
WARNING: no policy specified for oryx/127.2.182.66@KRBTEST.COM; defaulting to no policy
I20260928 21:54:35.515895 10491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:35.510055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:45217 (local address 127.2.182.65:55611)
0928 21:54:35.510210 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:35.510222 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:35.510285 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:35.510412 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:35.510413 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:35.510621 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:35.510688 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:35.510691 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:35.511302 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:35.511303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:35.512242 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:35.512245 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:35.512608 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:35.512609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:35.512677 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:35.513169 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:35.513174 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:35.513547 (+ 373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:35.514939 (+ 1392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:35.514940 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:35.514941 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:35.515152 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:35.515315 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:35.515316 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:35.515316 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:35.515394 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:35.515652 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:35.515653 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:35.515695 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 21:54:35.515743 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 21:54:35.515825 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":60,"threads_started":1}
I20260928 21:54:35.515944 10330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:35.510107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.65:55611 (local address 127.2.182.126:45217)
0928 21:54:35.510153 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:35.510155 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:35.510287 (+ 132us) server_negotiation.cc:407] Connection header received
0928 21:54:35.510337 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:35.510338 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:35.510352 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:35.510408 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:35.510759 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:35.511212 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:35.512338 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:35.512533 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:35.512592 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:35.513636 (+ 1044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:35.513637 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:35.513638 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:35.513643 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:35.514864 (+ 1221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:35.515208 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:35.515209 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:35.515209 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:35.515269 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:35.515497 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:35.515498 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:35.515498 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:35.515602 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:35.515653 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:35.515761 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 21:54:35.515815 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.182.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.182.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/krb5kdc/oryx_127.2.182.66.keytab.
Entry for principal oryx/127.2.182.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/krb5kdc/oryx_127.2.182.66.keytab.
I20260928 21:54:35.534693 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.66:0
--local_ip_for_outbound_sockets=127.2.182.66
--webserver_interface=127.2.182.66
--webserver_port=0
--tserver_master_addrs=127.2.182.126:45217
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/krb5kdc/oryx_127.2.182.66.keytab
--principal=oryx/127.2.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:36925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:54:35.616962 10493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:54:35.617205 10493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:54:35.617239 10493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:54:35.618433 10493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:54:35.618476 10493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:54:35.618505 10493 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 21:54:35.618528 10493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.66
I20260928 21:54:35.620080 10493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:36925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/krb5kdc/oryx_127.2.182.66.keytab
--principal=oryx/127.2.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.182.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.182.126:45217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.66
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:54:35.620325 10493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:54:35.620568 10493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:54:35.625639 10501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:35.625646 10493 server_base.cc:1034] running on GCE node
W20260928 21:54:35.625617 10498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:35.625615 10499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:35.626075 10493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:35.626302 10493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:35.627417 10493 hybrid_clock.cc:648] HybridClock initialized: now 1790632475627397 us; error 35 us; skew 500 ppm
Sep 28 21:54:35 dist-test-slave-k6lz krb5kdc[10236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632475, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:54:35.629796 10493 init.cc:377] Logged in from keytab as oryx/127.2.182.66@KRBTEST.COM (short username oryx)
I20260928 21:54:35.630249 10493 webserver.cc:460] Webserver started at http://127.2.182.66:36303/ using document root <none> and password file <none>
I20260928 21:54:35.630455 10493 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:35.630496 10493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:35.630620 10493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:54:35.631758 10493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/minicluster-data/ts-1/data/instance:
uuid: "05d1046a70f84d86a215b76dc85ef109"
format_stamp: "Formatted at 2026-09-28 21:54:35 on dist-test-slave-k6lz"
server_key: "c71c21f293e1262c6753f62c3d093c30"
server_key_iv: "562a6dd28f3ff421550cee65495aa779"
server_key_version: "encryptionkey@0"
I20260928 21:54:35.632081 10493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/minicluster-data/ts-1/wal/instance:
uuid: "05d1046a70f84d86a215b76dc85ef109"
format_stamp: "Formatted at 2026-09-28 21:54:35 on dist-test-slave-k6lz"
server_key: "c71c21f293e1262c6753f62c3d093c30"
server_key_iv: "562a6dd28f3ff421550cee65495aa779"
server_key_version: "encryptionkey@0"
I20260928 21:54:35.633385 10493 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:54:35.634210 10508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:35.634534 10493 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:54:35.634630 10493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/minicluster-data/ts-1/wal
uuid: "05d1046a70f84d86a215b76dc85ef109"
format_stamp: "Formatted at 2026-09-28 21:54:35 on dist-test-slave-k6lz"
server_key: "c71c21f293e1262c6753f62c3d093c30"
server_key_iv: "562a6dd28f3ff421550cee65495aa779"
server_key_version: "encryptionkey@0"
I20260928 21:54:35.634704 10493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:35.648150 10493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:54:35.650231 10493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:54:35.650370 10493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:54:35.650949 10493 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:54:35.651000 10493 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:35.651029 10493 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:54:35.651042 10493 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 21:54:35 dist-test-slave-k6lz krb5kdc[10236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632475, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.66@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM
I20260928 21:54:35.658120 10493 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.66:39201
I20260928 21:54:35.658421 10623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.66:39201 every 8 connection(s)
I20260928 21:54:35.658489 10493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
I20260928 21:54:35.659737 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 10493
I20260928 21:54:35.659837 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/minicluster-data/ts-1/wal/instance
I20260928 21:54:35.660040 2777 external_mini_cluster.cc:1428] Setting key ed360bd8b9cb0c064d79dc061723161a
I20260928 21:54:35.660953 10330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:35.651397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.66:60715 (local address 127.2.182.126:45217)
0928 21:54:35.651458 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:35.651461 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:35.651621 (+ 160us) server_negotiation.cc:407] Connection header received
0928 21:54:35.652091 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:35.652093 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:35.652106 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:35.652176 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:35.652840 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:35.653596 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:35.654710 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:35.654930 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:35.654997 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:35.657820 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:35.657823 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:35.657823 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:35.657832 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:35.659477 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:35.659936 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:35.659937 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:35.659938 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:35.660000 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:35.660259 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:35.660260 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:35.660260 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:35.660430 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:35.660488 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:35.660758 (+ 270us) server_negotiation.cc:299] Negotiation successful
0928 21:54:35.660842 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 21:54:35.661216 10517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:35.651306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:45217 (local address 127.2.182.66:60715)
0928 21:54:35.651503 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:35.651543 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:35.652010 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:35.652242 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:35.652245 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:35.652553 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:35.652740 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:35.652745 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:35.653703 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:35.653704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:35.654628 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:35.654632 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:35.655061 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:35.655062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:35.655225 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:35.655865 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:35.655875 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:35.657680 (+ 1805us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:35.659576 (+ 1896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:35.659582 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:35.659584 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:35.659842 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:35.660058 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:35.660059 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:35.660059 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:35.660197 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:35.660489 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:35.660501 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:35.660640 (+ 139us) client_negotiation.cc:769] Sending connection context
0928 21:54:35.660728 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 21:54:35.660974 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":116,"threads_started":1}
I20260928 21:54:35.662345 10624 heartbeater.cc:347] Connected to a master server at 127.2.182.126:45217
I20260928 21:54:35.662439 10624 heartbeater.cc:464] Registering TS with master...
I20260928 21:54:35.662601 10624 heartbeater.cc:511] Master 127.2.182.126:45217 requested a full tablet report, sending...
I20260928 21:54:35.662995 10281 ts_manager.cc:194] Registered new tserver with Master: 05d1046a70f84d86a215b76dc85ef109 (127.2.182.66:39201)
I20260928 21:54:35.663535 10281 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.182.66@KRBTEST.COM'} at 127.2.182.66:60715
WARNING: no policy specified for oryx/127.2.182.67@KRBTEST.COM; defaulting to no policy
I20260928 21:54:35.669842 10631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:35.663260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:45217 (local address 127.2.182.66:37883)
0928 21:54:35.663495 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:35.663505 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:35.663566 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:35.663753 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:35.663754 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:35.663968 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:35.664039 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:35.664041 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:35.664758 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:35.664759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:35.665603 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:35.665607 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:35.666047 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:35.666048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:35.666118 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:35.666596 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:35.666601 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:35.667004 (+ 403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:35.668707 (+ 1703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:35.668709 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:35.668709 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:35.668928 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:35.669148 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:35.669149 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:35.669150 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:35.669208 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:35.669494 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:35.669495 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:35.669539 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 21:54:35.669613 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 21:54:35.669706 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":179,"threads_started":1}
I20260928 21:54:35.669859 10330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:35.663310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.66:37883 (local address 127.2.182.126:45217)
0928 21:54:35.663373 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:35.663376 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:35.663572 (+ 196us) server_negotiation.cc:407] Connection header received
0928 21:54:35.663660 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:35.663661 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:35.663672 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:35.663697 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:35.664116 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:35.664658 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:35.665709 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:35.665968 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:35.666033 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:35.667117 (+ 1084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:35.667119 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:35.667120 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:35.667126 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:35.668627 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:35.669013 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:35.669014 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:35.669015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:35.669072 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:35.669263 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:35.669264 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:35.669264 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:35.669409 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:35.669462 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:35.669682 (+ 220us) server_negotiation.cc:299] Negotiation successful
0928 21:54:35.669755 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.182.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.182.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/krb5kdc/oryx_127.2.182.67.keytab.
Entry for principal oryx/127.2.182.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/krb5kdc/oryx_127.2.182.67.keytab.
I20260928 21:54:35.687297 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.67:0
--local_ip_for_outbound_sockets=127.2.182.67
--webserver_interface=127.2.182.67
--webserver_port=0
--tserver_master_addrs=127.2.182.126:45217
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/krb5kdc/oryx_127.2.182.67.keytab
--principal=oryx/127.2.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:36925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:54:35.767210 10633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:54:35.767486 10633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:54:35.767516 10633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:54:35.768698 10633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:54:35.768739 10633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:54:35.768759 10633 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 21:54:35.768775 10633 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.67
I20260928 21:54:35.770316 10633 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:36925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/krb5kdc/oryx_127.2.182.67.keytab
--principal=oryx/127.2.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.182.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.182.126:45217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10633
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.67
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:54:35.770550 10633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:54:35.770764 10633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:54:35.775632 10641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:35.775601 10638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:35.775596 10639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:35.775638 10633 server_base.cc:1034] running on GCE node
I20260928 21:54:35.775980 10633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:35.776167 10633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:35.777304 10633 hybrid_clock.cc:648] HybridClock initialized: now 1790632475777294 us; error 25 us; skew 500 ppm
Sep 28 21:54:35 dist-test-slave-k6lz krb5kdc[10236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632475, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:54:35.779860 10633 init.cc:377] Logged in from keytab as oryx/127.2.182.67@KRBTEST.COM (short username oryx)
I20260928 21:54:35.780269 10633 webserver.cc:460] Webserver started at http://127.2.182.67:37351/ using document root <none> and password file <none>
I20260928 21:54:35.780468 10633 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:35.780516 10633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:35.780650 10633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:54:35.781682 10633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/minicluster-data/ts-2/data/instance:
uuid: "c270fa5b12c2489abe94f87b4539e7bc"
format_stamp: "Formatted at 2026-09-28 21:54:35 on dist-test-slave-k6lz"
server_key: "b6cc29200f3cb7393bd4d2a6a6ed48eb"
server_key_iv: "efeb4fcf279dec98f540b97b3946ea8c"
server_key_version: "encryptionkey@0"
I20260928 21:54:35.782045 10633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/minicluster-data/ts-2/wal/instance:
uuid: "c270fa5b12c2489abe94f87b4539e7bc"
format_stamp: "Formatted at 2026-09-28 21:54:35 on dist-test-slave-k6lz"
server_key: "b6cc29200f3cb7393bd4d2a6a6ed48eb"
server_key_iv: "efeb4fcf279dec98f540b97b3946ea8c"
server_key_version: "encryptionkey@0"
I20260928 21:54:35.783283 10633 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:54:35.784011 10648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:35.784158 10633 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 21:54:35.784247 10633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/minicluster-data/ts-2/wal
uuid: "c270fa5b12c2489abe94f87b4539e7bc"
format_stamp: "Formatted at 2026-09-28 21:54:35 on dist-test-slave-k6lz"
server_key: "b6cc29200f3cb7393bd4d2a6a6ed48eb"
server_key_iv: "efeb4fcf279dec98f540b97b3946ea8c"
server_key_version: "encryptionkey@0"
I20260928 21:54:35.784307 10633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:35.825181 10633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:54:35.827450 10633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:54:35.827593 10633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:54:35.828109 10633 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:54:35.828218 10633 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:35.828316 10633 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:54:35.828344 10633 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 21:54:35 dist-test-slave-k6lz krb5kdc[10236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632475, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.67@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM
I20260928 21:54:35.835500 10633 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.67:44009
I20260928 21:54:35.835573 10763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.67:44009 every 8 connection(s)
I20260928 21:54:35.835857 10633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
I20260928 21:54:35.838212 10330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:35.828548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.67:51695 (local address 127.2.182.126:45217)
0928 21:54:35.828604 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:35.828606 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:35.828785 (+ 179us) server_negotiation.cc:407] Connection header received
0928 21:54:35.829343 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:35.829344 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:35.829357 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:35.829405 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:35.830239 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:35.831017 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:35.832573 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:35.832923 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:35.833007 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:35.835457 (+ 2450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:35.835459 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:35.835459 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:35.835468 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:35.836848 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:35.837291 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:35.837292 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:35.837293 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:35.837352 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:35.837580 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:35.837581 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:35.837581 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:35.837721 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:35.837779 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:35.838041 (+ 262us) server_negotiation.cc:299] Negotiation successful
0928 21:54:35.838122 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 21:54:35.838398 10657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:35.828524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:45217 (local address 127.2.182.67:51695)
0928 21:54:35.828674 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:35.828713 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:35.829240 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:35.829480 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:35.829485 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:35.829822 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:35.830118 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:35.830124 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:35.831129 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:35.831130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:35.832490 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:35.832495 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:35.833091 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:35.833092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:35.833283 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:35.833862 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:35.833871 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:35.835361 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:35.836931 (+ 1570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:35.836936 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:35.836939 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:35.837211 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:35.837409 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:35.837409 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:35.837410 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:35.837522 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:35.837770 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:35.837777 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:35.837919 (+ 142us) client_negotiation.cc:769] Sending connection context
0928 21:54:35.838010 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 21:54:35.838173 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":71,"threads_started":1}
I20260928 21:54:35.838937 10764 heartbeater.cc:347] Connected to a master server at 127.2.182.126:45217
I20260928 21:54:35.839025 10764 heartbeater.cc:464] Registering TS with master...
I20260928 21:54:35.839196 10764 heartbeater.cc:511] Master 127.2.182.126:45217 requested a full tablet report, sending...
I20260928 21:54:35.839670 10280 ts_manager.cc:194] Registered new tserver with Master: c270fa5b12c2489abe94f87b4539e7bc (127.2.182.67:44009)
I20260928 21:54:35.840175 10280 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.182.67@KRBTEST.COM'} at 127.2.182.67:51695
I20260928 21:54:35.842667 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 10633
I20260928 21:54:35.842767 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/minicluster-data/ts-2/wal/instance
I20260928 21:54:35.842947 2777 external_mini_cluster.cc:1428] Setting key 9ce6030a25169d1311fef88c8cc762c1
I20260928 21:54:35.843757 2777 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:54:35.846328 10770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:35.839882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:45217 (local address 127.2.182.67:44467)
0928 21:54:35.840009 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:35.840018 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:35.840066 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:35.840218 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:35.840219 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:35.840411 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:35.840497 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:35.840499 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:35.841394 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:35.841395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:35.842198 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:35.842200 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:35.842601 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:35.842602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:35.842687 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:35.843178 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:35.843183 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:35.843587 (+ 404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:35.845245 (+ 1658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:35.845246 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:35.845247 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:35.845490 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:35.845716 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:35.845717 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:35.845717 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:35.845770 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:35.846044 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:35.846045 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:35.846106 (+ 61us) client_negotiation.cc:769] Sending connection context
0928 21:54:35.846169 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 21:54:35.846253 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":67,"threads_started":1}
I20260928 21:54:35.846387 10330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:35.839942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.67:44467 (local address 127.2.182.126:45217)
0928 21:54:35.840012 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:35.840014 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:35.840062 (+ 48us) server_negotiation.cc:407] Connection header received
0928 21:54:35.840112 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:35.840113 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:35.840142 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:35.840200 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:35.840578 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:35.841302 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:35.842273 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:35.842520 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:35.842572 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:35.843666 (+ 1094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:35.843668 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:35.843668 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:35.843675 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:35.845154 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:35.845581 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:35.845583 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:35.845583 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:35.845648 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:35.845830 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:35.845831 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:35.845832 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:35.845979 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:35.846048 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:35.846205 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 21:54:35.846297 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Sep 28 21:54:35 dist-test-slave-k6lz krb5kdc[10236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632475, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 21:54:35 dist-test-slave-k6lz krb5kdc[10236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:54:35 dist-test-slave-k6lz krb5kdc[10236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
W20260928 21:54:35.865108 10330 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:54:35.859420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43110 (local address 127.2.182.126:45217)
0928 21:54:35.859480 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:35.859482 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:35.859491 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:54:35.859548 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:35.859549 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:35.859559 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:35.859623 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:35.860227 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:35.860939 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:35.861759 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:35.861952 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:35.862013 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:35.865033 (+ 3020us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43110: BlockingRecv error: recv got EOF from 127.0.0.1:43110 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 28 21:54:35 dist-test-slave-k6lz krb5kdc[10236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632475, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM
I20260928 21:54:35.872673 10330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:35.865955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43112 (local address 127.2.182.126:45217)
0928 21:54:35.866016 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:35.866018 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:35.866079 (+ 61us) server_negotiation.cc:407] Connection header received
0928 21:54:35.866141 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:35.866142 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:35.866153 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:35.866198 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:35.866637 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:35.867130 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:35.867838 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:35.868021 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:35.868076 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:35.870282 (+ 2206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:35.870283 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:35.870284 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:35.870291 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:35.871567 (+ 1276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:35.871914 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:35.871915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:35.871915 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:35.871966 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:35.872202 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:35.872203 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:35.872203 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:35.872332 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:35.872424 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:35.872541 (+ 117us) server_negotiation.cc:299] Negotiation successful
0928 21:54:35.872604 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 21:54:35.874406 10280 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43112:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:54:35.876034 10280 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:54:35.882370 10787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:35.878370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.66:39201 (local address 127.0.0.1:34258)
0928 21:54:35.878678 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:35.878720 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:35.878787 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:35.879015 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:35.879018 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:35.879020 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:54:35.879095 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:35.879097 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:35.880142 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:35.880143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:35.881313 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:35.881315 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:35.882077 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:35.882077 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:35.882159 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:35.882173 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 21:54:35.882237 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 21:54:35.882288 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":59,"threads_started":1}
I20260928 21:54:35.882383 10789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:35.878522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34258 (local address 127.2.182.66:39201)
0928 21:54:35.878673 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:35.878677 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:35.878864 (+ 187us) server_negotiation.cc:407] Connection header received
0928 21:54:35.878903 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:35.878905 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:35.878920 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:35.878998 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:54:35.879176 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:35.880064 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:35.881397 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:35.882005 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:35.882066 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:35.882145 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:35.882249 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 21:54:35.882288 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":62,"threads_started":1}
I20260928 21:54:35.882812 10790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:35.878565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.65:35485 (local address 127.0.0.1:56518)
0928 21:54:35.878983 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:35.878993 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:35.879040 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:35.879371 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:35.879372 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:35.879374 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:54:35.879409 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:35.879410 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:35.880586 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:35.880587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:35.881541 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:35.881543 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:35.882518 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:35.882519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:35.882585 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:35.882594 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 21:54:35.882638 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 21:54:35.882688 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"mutex_wait_us":12,"thread_start_us":56,"threads_started":1}
I20260928 21:54:35.882893 10788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:35.878472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40838 (local address 127.2.182.67:44009)
0928 21:54:35.878763 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:35.878766 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:35.878777 (+ 11us) server_negotiation.cc:407] Connection header received
0928 21:54:35.878872 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:35.878874 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:35.878952 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:35.879086 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:54:35.879290 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:35.880432 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:35.881643 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:35.882346 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:35.882412 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:35.882484 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:35.882771 (+ 287us) server_negotiation.cc:299] Negotiation successful
0928 21:54:35.882809 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":87,"threads_started":1}
I20260928 21:54:35.882880 10791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:35.878694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56518 (local address 127.2.182.65:35485)
0928 21:54:35.878935 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:35.878938 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:35.879157 (+ 219us) server_negotiation.cc:407] Connection header received
0928 21:54:35.879206 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:35.879209 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:35.879224 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:35.879291 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:54:35.879471 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:35.880516 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:35.881622 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:35.882345 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:35.882412 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:35.882481 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:35.882730 (+ 249us) server_negotiation.cc:299] Negotiation successful
0928 21:54:35.882770 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":77,"threads_started":1}
I20260928 21:54:35.883062 10786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:35.878393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.67:44009 (local address 127.0.0.1:40838)
0928 21:54:35.878679 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:35.878719 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:35.878785 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:35.879089 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:35.879090 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:35.879093 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:54:35.879216 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:35.879218 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:35.880533 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:35.880534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:35.881565 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:35.881567 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:35.882517 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:35.882518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:35.882585 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:35.882592 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 21:54:35.882638 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 21:54:35.882687 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":29,"thread_start_us":69,"threads_started":1}
I20260928 21:54:35.883359 10558 tablet_service.cc:1467] Processing CreateTablet for tablet 292d18401c544deeb0d5d814a64e4dab (DEFAULT_TABLE table=test-table [id=2eec63287db64d0db25d725a884e44f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:54:35.883694 10558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 292d18401c544deeb0d5d814a64e4dab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:54:35.883745 10418 tablet_service.cc:1467] Processing CreateTablet for tablet 292d18401c544deeb0d5d814a64e4dab (DEFAULT_TABLE table=test-table [id=2eec63287db64d0db25d725a884e44f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:54:35.884020 10418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 292d18401c544deeb0d5d814a64e4dab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:54:35.886582 10792 tablet_bootstrap.cc:492] T 292d18401c544deeb0d5d814a64e4dab P 05d1046a70f84d86a215b76dc85ef109: Bootstrap starting.
I20260928 21:54:35.886941 10793 tablet_bootstrap.cc:492] T 292d18401c544deeb0d5d814a64e4dab P 8c5ce1c5850d406fbd3f4eeca6fde124: Bootstrap starting.
I20260928 21:54:35.887099 10698 tablet_service.cc:1467] Processing CreateTablet for tablet 292d18401c544deeb0d5d814a64e4dab (DEFAULT_TABLE table=test-table [id=2eec63287db64d0db25d725a884e44f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:54:35.887280 10792 tablet_bootstrap.cc:654] T 292d18401c544deeb0d5d814a64e4dab P 05d1046a70f84d86a215b76dc85ef109: Neither blocks nor log segments found. Creating new log.
I20260928 21:54:35.887446 10698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 292d18401c544deeb0d5d814a64e4dab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:54:35.887521 10793 tablet_bootstrap.cc:654] T 292d18401c544deeb0d5d814a64e4dab P 8c5ce1c5850d406fbd3f4eeca6fde124: Neither blocks nor log segments found. Creating new log.
I20260928 21:54:35.887610 10792 log.cc:824] T 292d18401c544deeb0d5d814a64e4dab P 05d1046a70f84d86a215b76dc85ef109: Log is configured to *not* fsync() on all Append() calls
I20260928 21:54:35.887830 10793 log.cc:824] T 292d18401c544deeb0d5d814a64e4dab P 8c5ce1c5850d406fbd3f4eeca6fde124: Log is configured to *not* fsync() on all Append() calls
I20260928 21:54:35.888581 10793 tablet_bootstrap.cc:492] T 292d18401c544deeb0d5d814a64e4dab P 8c5ce1c5850d406fbd3f4eeca6fde124: No bootstrap required, opened a new log
I20260928 21:54:35.888648 10793 ts_tablet_manager.cc:1397] T 292d18401c544deeb0d5d814a64e4dab P 8c5ce1c5850d406fbd3f4eeca6fde124: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:54:35.889206 10792 tablet_bootstrap.cc:492] T 292d18401c544deeb0d5d814a64e4dab P 05d1046a70f84d86a215b76dc85ef109: No bootstrap required, opened a new log
I20260928 21:54:35.889276 10792 ts_tablet_manager.cc:1397] T 292d18401c544deeb0d5d814a64e4dab P 05d1046a70f84d86a215b76dc85ef109: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 21:54:35.890241 10793 raft_consensus.cc:348] T 292d18401c544deeb0d5d814a64e4dab P 8c5ce1c5850d406fbd3f4eeca6fde124 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c270fa5b12c2489abe94f87b4539e7bc" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 44009 } } peers { permanent_uuid: "05d1046a70f84d86a215b76dc85ef109" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 39201 } } peers { permanent_uuid: "8c5ce1c5850d406fbd3f4eeca6fde124" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35485 } }
I20260928 21:54:35.890369 10793 raft_consensus.cc:374] T 292d18401c544deeb0d5d814a64e4dab P 8c5ce1c5850d406fbd3f4eeca6fde124 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:54:35.890409 10793 raft_consensus.cc:729] T 292d18401c544deeb0d5d814a64e4dab P 8c5ce1c5850d406fbd3f4eeca6fde124 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c5ce1c5850d406fbd3f4eeca6fde124, State: Initialized, Role: FOLLOWER
I20260928 21:54:35.890538 10793 consensus_queue.cc:260] T 292d18401c544deeb0d5d814a64e4dab P 8c5ce1c5850d406fbd3f4eeca6fde124 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c270fa5b12c2489abe94f87b4539e7bc" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 44009 } } peers { permanent_uuid: "05d1046a70f84d86a215b76dc85ef109" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 39201 } } peers { permanent_uuid: "8c5ce1c5850d406fbd3f4eeca6fde124" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35485 } }
I20260928 21:54:35.890791 10793 ts_tablet_manager.cc:1428] T 292d18401c544deeb0d5d814a64e4dab P 8c5ce1c5850d406fbd3f4eeca6fde124: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 21:54:35.890794 10796 tablet_bootstrap.cc:492] T 292d18401c544deeb0d5d814a64e4dab P c270fa5b12c2489abe94f87b4539e7bc: Bootstrap starting.
I20260928 21:54:35.890914 10484 heartbeater.cc:503] Master 127.2.182.126:45217 was elected leader, sending a full tablet report...
I20260928 21:54:35.891186 10792 raft_consensus.cc:348] T 292d18401c544deeb0d5d814a64e4dab P 05d1046a70f84d86a215b76dc85ef109 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c270fa5b12c2489abe94f87b4539e7bc" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 44009 } } peers { permanent_uuid: "05d1046a70f84d86a215b76dc85ef109" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 39201 } } peers { permanent_uuid: "8c5ce1c5850d406fbd3f4eeca6fde124" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35485 } }
I20260928 21:54:35.891357 10792 raft_consensus.cc:374] T 292d18401c544deeb0d5d814a64e4dab P 05d1046a70f84d86a215b76dc85ef109 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:54:35.891407 10792 raft_consensus.cc:729] T 292d18401c544deeb0d5d814a64e4dab P 05d1046a70f84d86a215b76dc85ef109 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05d1046a70f84d86a215b76dc85ef109, State: Initialized, Role: FOLLOWER
I20260928 21:54:35.891471 10796 tablet_bootstrap.cc:654] T 292d18401c544deeb0d5d814a64e4dab P c270fa5b12c2489abe94f87b4539e7bc: Neither blocks nor log segments found. Creating new log.
I20260928 21:54:35.891515 10792 consensus_queue.cc:260] T 292d18401c544deeb0d5d814a64e4dab P 05d1046a70f84d86a215b76dc85ef109 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c270fa5b12c2489abe94f87b4539e7bc" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 44009 } } peers { permanent_uuid: "05d1046a70f84d86a215b76dc85ef109" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 39201 } } peers { permanent_uuid: "8c5ce1c5850d406fbd3f4eeca6fde124" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35485 } }
I20260928 21:54:35.891755 10792 ts_tablet_manager.cc:1428] T 292d18401c544deeb0d5d814a64e4dab P 05d1046a70f84d86a215b76dc85ef109: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260928 21:54:35.891767 10796 log.cc:824] T 292d18401c544deeb0d5d814a64e4dab P c270fa5b12c2489abe94f87b4539e7bc: Log is configured to *not* fsync() on all Append() calls
I20260928 21:54:35.891901 10624 heartbeater.cc:503] Master 127.2.182.126:45217 was elected leader, sending a full tablet report...
I20260928 21:54:35.892488 10796 tablet_bootstrap.cc:492] T 292d18401c544deeb0d5d814a64e4dab P c270fa5b12c2489abe94f87b4539e7bc: No bootstrap required, opened a new log
I20260928 21:54:35.892554 10796 ts_tablet_manager.cc:1397] T 292d18401c544deeb0d5d814a64e4dab P c270fa5b12c2489abe94f87b4539e7bc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:54:35.894098 10796 raft_consensus.cc:348] T 292d18401c544deeb0d5d814a64e4dab P c270fa5b12c2489abe94f87b4539e7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c270fa5b12c2489abe94f87b4539e7bc" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 44009 } } peers { permanent_uuid: "05d1046a70f84d86a215b76dc85ef109" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 39201 } } peers { permanent_uuid: "8c5ce1c5850d406fbd3f4eeca6fde124" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35485 } }
I20260928 21:54:35.894233 10796 raft_consensus.cc:374] T 292d18401c544deeb0d5d814a64e4dab P c270fa5b12c2489abe94f87b4539e7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:54:35.894254 10796 raft_consensus.cc:729] T 292d18401c544deeb0d5d814a64e4dab P c270fa5b12c2489abe94f87b4539e7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c270fa5b12c2489abe94f87b4539e7bc, State: Initialized, Role: FOLLOWER
I20260928 21:54:35.894325 10796 consensus_queue.cc:260] T 292d18401c544deeb0d5d814a64e4dab P c270fa5b12c2489abe94f87b4539e7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c270fa5b12c2489abe94f87b4539e7bc" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 44009 } } peers { permanent_uuid: "05d1046a70f84d86a215b76dc85ef109" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 39201 } } peers { permanent_uuid: "8c5ce1c5850d406fbd3f4eeca6fde124" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35485 } }
I20260928 21:54:35.894517 10796 ts_tablet_manager.cc:1428] T 292d18401c544deeb0d5d814a64e4dab P c270fa5b12c2489abe94f87b4539e7bc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 21:54:35.894588 10764 heartbeater.cc:503] Master 127.2.182.126:45217 was elected leader, sending a full tablet report...
I20260928 21:54:35.901652 10800 raft_consensus.cc:482] T 292d18401c544deeb0d5d814a64e4dab P c270fa5b12c2489abe94f87b4539e7bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:54:35.901774 10800 raft_consensus.cc:504] T 292d18401c544deeb0d5d814a64e4dab P c270fa5b12c2489abe94f87b4539e7bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c270fa5b12c2489abe94f87b4539e7bc" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 44009 } } peers { permanent_uuid: "05d1046a70f84d86a215b76dc85ef109" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 39201 } } peers { permanent_uuid: "8c5ce1c5850d406fbd3f4eeca6fde124" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35485 } }
I20260928 21:54:35.902117 10800 leader_election.cc:288] T 292d18401c544deeb0d5d814a64e4dab P c270fa5b12c2489abe94f87b4539e7bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05d1046a70f84d86a215b76dc85ef109 (127.2.182.66:39201), 8c5ce1c5850d406fbd3f4eeca6fde124 (127.2.182.65:35485)
I20260928 21:54:35.905072 10789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:35.902507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.67:52753 (local address 127.2.182.66:39201)
0928 21:54:35.902568 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:35.902571 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:35.902579 (+ 8us) server_negotiation.cc:407] Connection header received
0928 21:54:35.902608 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:35.902608 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:35.902618 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:35.902675 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:54:35.902818 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:35.903421 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:35.904390 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:35.904768 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:35.904830 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:35.904839 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:35.904970 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 21:54:35.905003 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 21:54:35.905072 10791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:35.902406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.67:36501 (local address 127.2.182.65:35485)
0928 21:54:35.902455 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:35.902457 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:35.902549 (+ 92us) server_negotiation.cc:407] Connection header received
0928 21:54:35.902633 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:35.902634 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:35.902643 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:35.902684 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:54:35.902859 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:35.903433 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:35.904422 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:35.904773 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:35.904830 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:35.904839 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:35.904965 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 21:54:35.905001 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 21:54:35.905094 10801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:35.902305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.66:39201 (local address 127.2.182.67:52753)
0928 21:54:35.902500 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:35.902508 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:35.902558 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:35.902681 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:35.902682 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:35.902684 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:54:35.902756 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:35.902758 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:35.903498 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:35.903499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:35.904310 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:35.904312 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:35.904836 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:35.904837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:35.904915 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:35.904918 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:54:35.904962 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 21:54:35.904999 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":20,"spinlock_wait_cycles":9600,"thread_start_us":114,"threads_started":1}
I20260928 21:54:35.905100 10657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:35.902291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.65:35485 (local address 127.2.182.67:36501)
0928 21:54:35.902477 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:35.902489 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:35.902553 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:35.902729 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:35.902730 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:35.902732 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:54:35.902800 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:35.902802 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:35.903498 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:35.903499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:35.904340 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:35.904342 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:35.904836 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:35.904837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:35.904908 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:35.904911 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:54:35.904954 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 21:54:35.905001 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":43}
I20260928 21:54:35.905414 10438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292d18401c544deeb0d5d814a64e4dab" candidate_uuid: "c270fa5b12c2489abe94f87b4539e7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c5ce1c5850d406fbd3f4eeca6fde124" is_pre_election: true
I20260928 21:54:35.905560 10438 raft_consensus.cc:2454] T 292d18401c544deeb0d5d814a64e4dab P 8c5ce1c5850d406fbd3f4eeca6fde124 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c270fa5b12c2489abe94f87b4539e7bc in term 0.
I20260928 21:54:35.905522 10576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292d18401c544deeb0d5d814a64e4dab" candidate_uuid: "c270fa5b12c2489abe94f87b4539e7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05d1046a70f84d86a215b76dc85ef109" is_pre_election: true
I20260928 21:54:35.905650 10576 raft_consensus.cc:2454] T 292d18401c544deeb0d5d814a64e4dab P 05d1046a70f84d86a215b76dc85ef109 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c270fa5b12c2489abe94f87b4539e7bc in term 0.
I20260928 21:54:35.905768 10650 leader_election.cc:302] T 292d18401c544deeb0d5d814a64e4dab P c270fa5b12c2489abe94f87b4539e7bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c5ce1c5850d406fbd3f4eeca6fde124, c270fa5b12c2489abe94f87b4539e7bc; no voters:
I20260928 21:54:35.905874 10800 raft_consensus.cc:2781] T 292d18401c544deeb0d5d814a64e4dab P c270fa5b12c2489abe94f87b4539e7bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:54:35.905932 10800 raft_consensus.cc:482] T 292d18401c544deeb0d5d814a64e4dab P c270fa5b12c2489abe94f87b4539e7bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:54:35.905948 10800 raft_consensus.cc:3037] T 292d18401c544deeb0d5d814a64e4dab P c270fa5b12c2489abe94f87b4539e7bc [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:54:35.906625 10800 raft_consensus.cc:504] T 292d18401c544deeb0d5d814a64e4dab P c270fa5b12c2489abe94f87b4539e7bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c270fa5b12c2489abe94f87b4539e7bc" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 44009 } } peers { permanent_uuid: "05d1046a70f84d86a215b76dc85ef109" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 39201 } } peers { permanent_uuid: "8c5ce1c5850d406fbd3f4eeca6fde124" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35485 } }
I20260928 21:54:35.906749 10800 leader_election.cc:288] T 292d18401c544deeb0d5d814a64e4dab P c270fa5b12c2489abe94f87b4539e7bc [CANDIDATE]: Term 1 election: Requested vote from peers 05d1046a70f84d86a215b76dc85ef109 (127.2.182.66:39201), 8c5ce1c5850d406fbd3f4eeca6fde124 (127.2.182.65:35485)
I20260928 21:54:35.906919 10576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292d18401c544deeb0d5d814a64e4dab" candidate_uuid: "c270fa5b12c2489abe94f87b4539e7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05d1046a70f84d86a215b76dc85ef109"
I20260928 21:54:35.907011 10576 raft_consensus.cc:3037] T 292d18401c544deeb0d5d814a64e4dab P 05d1046a70f84d86a215b76dc85ef109 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:54:35.907073 10438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292d18401c544deeb0d5d814a64e4dab" candidate_uuid: "c270fa5b12c2489abe94f87b4539e7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c5ce1c5850d406fbd3f4eeca6fde124"
I20260928 21:54:35.907150 10438 raft_consensus.cc:3037] T 292d18401c544deeb0d5d814a64e4dab P 8c5ce1c5850d406fbd3f4eeca6fde124 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:54:35.907922 10576 raft_consensus.cc:2454] T 292d18401c544deeb0d5d814a64e4dab P 05d1046a70f84d86a215b76dc85ef109 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c270fa5b12c2489abe94f87b4539e7bc in term 1.
I20260928 21:54:35.908043 10438 raft_consensus.cc:2454] T 292d18401c544deeb0d5d814a64e4dab P 8c5ce1c5850d406fbd3f4eeca6fde124 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c270fa5b12c2489abe94f87b4539e7bc in term 1.
I20260928 21:54:35.908438 10651 leader_election.cc:302] T 292d18401c544deeb0d5d814a64e4dab P c270fa5b12c2489abe94f87b4539e7bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05d1046a70f84d86a215b76dc85ef109, c270fa5b12c2489abe94f87b4539e7bc; no voters:
I20260928 21:54:35.908567 10800 raft_consensus.cc:2781] T 292d18401c544deeb0d5d814a64e4dab P c270fa5b12c2489abe94f87b4539e7bc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:54:35.908805 10800 raft_consensus.cc:686] T 292d18401c544deeb0d5d814a64e4dab P c270fa5b12c2489abe94f87b4539e7bc [term 1 LEADER]: Becoming Leader. State: Replica: c270fa5b12c2489abe94f87b4539e7bc, State: Running, Role: LEADER
I20260928 21:54:35.908908 10800 consensus_queue.cc:237] T 292d18401c544deeb0d5d814a64e4dab P c270fa5b12c2489abe94f87b4539e7bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c270fa5b12c2489abe94f87b4539e7bc" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 44009 } } peers { permanent_uuid: "05d1046a70f84d86a215b76dc85ef109" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 39201 } } peers { permanent_uuid: "8c5ce1c5850d406fbd3f4eeca6fde124" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35485 } }
W20260928 21:54:35.909174 10625 tablet.cc:2369] T 292d18401c544deeb0d5d814a64e4dab P 05d1046a70f84d86a215b76dc85ef109: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:54:35.909735 10280 catalog_manager.cc:5495] T 292d18401c544deeb0d5d814a64e4dab P c270fa5b12c2489abe94f87b4539e7bc reported cstate change: term changed from 0 to 1, leader changed from <none> to c270fa5b12c2489abe94f87b4539e7bc (127.2.182.67). New cstate: current_term: 1 leader_uuid: "c270fa5b12c2489abe94f87b4539e7bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c270fa5b12c2489abe94f87b4539e7bc" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 44009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05d1046a70f84d86a215b76dc85ef109" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 39201 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c5ce1c5850d406fbd3f4eeca6fde124" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35485 } health_report { overall_health: UNKNOWN } } }
I20260928 21:54:35.921921 10788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:35.919526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40850 (local address 127.2.182.67:44009)
0928 21:54:35.919568 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:35.919570 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:35.919579 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:54:35.919654 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:35.919655 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:35.919672 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:35.919731 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:54:35.919836 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:35.920329 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:35.921076 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:35.921276 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:35.921335 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:35.921523 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:54:35.921628 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:54:35.921725 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:35.921738 (+ 13us) server_negotiation.cc:299] Negotiation successful
0928 21:54:35.921757 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 21:54:35.923728 10576 raft_consensus.cc:1261] T 292d18401c544deeb0d5d814a64e4dab P 05d1046a70f84d86a215b76dc85ef109 [term 1 FOLLOWER]: Refusing update from remote peer c270fa5b12c2489abe94f87b4539e7bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:54:35.923738 10438 raft_consensus.cc:1261] T 292d18401c544deeb0d5d814a64e4dab P 8c5ce1c5850d406fbd3f4eeca6fde124 [term 1 FOLLOWER]: Refusing update from remote peer c270fa5b12c2489abe94f87b4539e7bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:54:35.924039 10800 consensus_queue.cc:1035] T 292d18401c544deeb0d5d814a64e4dab P c270fa5b12c2489abe94f87b4539e7bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c5ce1c5850d406fbd3f4eeca6fde124" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:54:35.924181 10802 consensus_queue.cc:1035] T 292d18401c544deeb0d5d814a64e4dab P c270fa5b12c2489abe94f87b4539e7bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "05d1046a70f84d86a215b76dc85ef109" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 39201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:54:35.926505 10805 mvcc.cc:205] Tried to move back new op lower bound from 7334430621381029888 to 7334430621323694080. Current Snapshot: MvccSnapshot[applied={T|T < 7334430621381029888}]
I20260928 21:54:35.926716 10807 mvcc.cc:205] Tried to move back new op lower bound from 7334430621381029888 to 7334430621323694080. Current Snapshot: MvccSnapshot[applied={T|T < 7334430621381029888}]
I20260928 21:54:35.926982 10806 mvcc.cc:205] Tried to move back new op lower bound from 7334430621381029888 to 7334430621323694080. Current Snapshot: MvccSnapshot[applied={T|T < 7334430621381029888}]
I20260928 21:54:35.928371 10281 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43112:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:54:35.928471 10281 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43112:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:54:35.929230 10281 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 85cac027afc849588f7eae9726ee1eb1: Sending DeleteTablet for 3 replicas of tablet 292d18401c544deeb0d5d814a64e4dab
I20260928 21:54:35.929564 10698 tablet_service.cc:1514] Processing DeleteTablet for tablet 292d18401c544deeb0d5d814a64e4dab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:54:35 UTC) from {username='oryx', principal='oryx/127.2.182.126@KRBTEST.COM'} at 127.0.0.1:40838
I20260928 21:54:35.929747 10558 tablet_service.cc:1514] Processing DeleteTablet for tablet 292d18401c544deeb0d5d814a64e4dab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:54:35 UTC) from {username='oryx', principal='oryx/127.2.182.126@KRBTEST.COM'} at 127.0.0.1:34258
I20260928 21:54:35.930011 10816 tablet_replica.cc:330] T 292d18401c544deeb0d5d814a64e4dab P 05d1046a70f84d86a215b76dc85ef109: stopping tablet replica
I20260928 21:54:35.929998 10418 tablet_service.cc:1514] Processing DeleteTablet for tablet 292d18401c544deeb0d5d814a64e4dab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:54:35 UTC) from {username='oryx', principal='oryx/127.2.182.126@KRBTEST.COM'} at 127.0.0.1:56518
I20260928 21:54:35.930140 10816 raft_consensus.cc:2229] T 292d18401c544deeb0d5d814a64e4dab P 05d1046a70f84d86a215b76dc85ef109 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:54:35.930261 10816 raft_consensus.cc:2258] T 292d18401c544deeb0d5d814a64e4dab P 05d1046a70f84d86a215b76dc85ef109 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:54:35.930501 10817 tablet_replica.cc:330] T 292d18401c544deeb0d5d814a64e4dab P 8c5ce1c5850d406fbd3f4eeca6fde124: stopping tablet replica
I20260928 21:54:35.930497 10815 tablet_replica.cc:330] T 292d18401c544deeb0d5d814a64e4dab P c270fa5b12c2489abe94f87b4539e7bc: stopping tablet replica
I20260928 21:54:35.930470 10281 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43112:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 21:54:35.930591 10815 raft_consensus.cc:2229] T 292d18401c544deeb0d5d814a64e4dab P c270fa5b12c2489abe94f87b4539e7bc [term 1 LEADER]: Raft consensus shutting down.
W20260928 21:54:35.930619 10281 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:54:35.930680 10815 raft_consensus.cc:2258] T 292d18401c544deeb0d5d814a64e4dab P c270fa5b12c2489abe94f87b4539e7bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:54:35.931147 10815 ts_tablet_manager.cc:1905] T 292d18401c544deeb0d5d814a64e4dab P c270fa5b12c2489abe94f87b4539e7bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:54:35.931154 10816 ts_tablet_manager.cc:1905] T 292d18401c544deeb0d5d814a64e4dab P 05d1046a70f84d86a215b76dc85ef109: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:54:35.931564 10817 raft_consensus.cc:2229] T 292d18401c544deeb0d5d814a64e4dab P 8c5ce1c5850d406fbd3f4eeca6fde124 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:54:35.931744 10817 raft_consensus.cc:2258] T 292d18401c544deeb0d5d814a64e4dab P 8c5ce1c5850d406fbd3f4eeca6fde124 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:54:35.932003 10698 tablet_service.cc:1467] Processing CreateTablet for tablet 3542c748bdb34c55a8ef129bebeee9bf (DEFAULT_TABLE table=test-table [id=a243e97f7adc49529e99154b5ea30298]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:54:35.932044 10418 tablet_service.cc:1467] Processing CreateTablet for tablet 3542c748bdb34c55a8ef129bebeee9bf (DEFAULT_TABLE table=test-table [id=a243e97f7adc49529e99154b5ea30298]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:54:35.932120 10698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3542c748bdb34c55a8ef129bebeee9bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:54:35.932125 10418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3542c748bdb34c55a8ef129bebeee9bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:54:35.933357 10793 tablet_bootstrap.cc:492] T 3542c748bdb34c55a8ef129bebeee9bf P 8c5ce1c5850d406fbd3f4eeca6fde124: Bootstrap starting.
I20260928 21:54:35.933475 10558 tablet_service.cc:1467] Processing CreateTablet for tablet 3542c748bdb34c55a8ef129bebeee9bf (DEFAULT_TABLE table=test-table [id=a243e97f7adc49529e99154b5ea30298]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:54:35.933584 10558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3542c748bdb34c55a8ef129bebeee9bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:54:35.933800 10793 tablet_bootstrap.cc:654] T 3542c748bdb34c55a8ef129bebeee9bf P 8c5ce1c5850d406fbd3f4eeca6fde124: Neither blocks nor log segments found. Creating new log.
I20260928 21:54:35.934123 10796 tablet_bootstrap.cc:492] T 3542c748bdb34c55a8ef129bebeee9bf P c270fa5b12c2489abe94f87b4539e7bc: Bootstrap starting.
I20260928 21:54:35.934419 10793 tablet_bootstrap.cc:492] T 3542c748bdb34c55a8ef129bebeee9bf P 8c5ce1c5850d406fbd3f4eeca6fde124: No bootstrap required, opened a new log
I20260928 21:54:35.934461 10793 ts_tablet_manager.cc:1397] T 3542c748bdb34c55a8ef129bebeee9bf P 8c5ce1c5850d406fbd3f4eeca6fde124: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:54:35.934527 10796 tablet_bootstrap.cc:654] T 3542c748bdb34c55a8ef129bebeee9bf P c270fa5b12c2489abe94f87b4539e7bc: Neither blocks nor log segments found. Creating new log.
I20260928 21:54:35.934597 10793 raft_consensus.cc:348] T 3542c748bdb34c55a8ef129bebeee9bf P 8c5ce1c5850d406fbd3f4eeca6fde124 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c5ce1c5850d406fbd3f4eeca6fde124" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35485 } } peers { permanent_uuid: "05d1046a70f84d86a215b76dc85ef109" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 39201 } } peers { permanent_uuid: "c270fa5b12c2489abe94f87b4539e7bc" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 44009 } }
I20260928 21:54:35.934661 10793 raft_consensus.cc:374] T 3542c748bdb34c55a8ef129bebeee9bf P 8c5ce1c5850d406fbd3f4eeca6fde124 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:54:35.934679 10793 raft_consensus.cc:729] T 3542c748bdb34c55a8ef129bebeee9bf P 8c5ce1c5850d406fbd3f4eeca6fde124 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c5ce1c5850d406fbd3f4eeca6fde124, State: Initialized, Role: FOLLOWER
I20260928 21:54:35.934721 10793 consensus_queue.cc:260] T 3542c748bdb34c55a8ef129bebeee9bf P 8c5ce1c5850d406fbd3f4eeca6fde124 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c5ce1c5850d406fbd3f4eeca6fde124" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35485 } } peers { permanent_uuid: "05d1046a70f84d86a215b76dc85ef109" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 39201 } } peers { permanent_uuid: "c270fa5b12c2489abe94f87b4539e7bc" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 44009 } }
I20260928 21:54:35.934798 10793 ts_tablet_manager.cc:1428] T 3542c748bdb34c55a8ef129bebeee9bf P 8c5ce1c5850d406fbd3f4eeca6fde124: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:35.934813 10792 tablet_bootstrap.cc:492] T 3542c748bdb34c55a8ef129bebeee9bf P 05d1046a70f84d86a215b76dc85ef109: Bootstrap starting.
I20260928 21:54:35.935199 10792 tablet_bootstrap.cc:654] T 3542c748bdb34c55a8ef129bebeee9bf P 05d1046a70f84d86a215b76dc85ef109: Neither blocks nor log segments found. Creating new log.
I20260928 21:54:35.935746 10815 ts_tablet_manager.cc:1918] T 292d18401c544deeb0d5d814a64e4dab P c270fa5b12c2489abe94f87b4539e7bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:54:35.935827 10815 log.cc:1196] T 292d18401c544deeb0d5d814a64e4dab P c270fa5b12c2489abe94f87b4539e7bc: Deleting WAL directory at /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/minicluster-data/ts-2/wal/wals/292d18401c544deeb0d5d814a64e4dab
I20260928 21:54:35.936079 10815 ts_tablet_manager.cc:1939] T 292d18401c544deeb0d5d814a64e4dab P c270fa5b12c2489abe94f87b4539e7bc: Deleting consensus metadata
I20260928 21:54:35.936432 10817 ts_tablet_manager.cc:1905] T 292d18401c544deeb0d5d814a64e4dab P 8c5ce1c5850d406fbd3f4eeca6fde124: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:54:35.937078 10266 catalog_manager.cc:4841] TS c270fa5b12c2489abe94f87b4539e7bc (127.2.182.67:44009): tablet 292d18401c544deeb0d5d814a64e4dab (table test-table [id=2eec63287db64d0db25d725a884e44f0]) successfully deleted
I20260928 21:54:35.937678 10792 tablet_bootstrap.cc:492] T 3542c748bdb34c55a8ef129bebeee9bf P 05d1046a70f84d86a215b76dc85ef109: No bootstrap required, opened a new log
I20260928 21:54:35.937736 10792 ts_tablet_manager.cc:1397] T 3542c748bdb34c55a8ef129bebeee9bf P 05d1046a70f84d86a215b76dc85ef109: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 21:54:35.937826 10816 ts_tablet_manager.cc:1918] T 292d18401c544deeb0d5d814a64e4dab P 05d1046a70f84d86a215b76dc85ef109: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:54:35.937897 10816 log.cc:1196] T 292d18401c544deeb0d5d814a64e4dab P 05d1046a70f84d86a215b76dc85ef109: Deleting WAL directory at /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/minicluster-data/ts-1/wal/wals/292d18401c544deeb0d5d814a64e4dab
I20260928 21:54:35.937877 10792 raft_consensus.cc:348] T 3542c748bdb34c55a8ef129bebeee9bf P 05d1046a70f84d86a215b76dc85ef109 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c5ce1c5850d406fbd3f4eeca6fde124" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35485 } } peers { permanent_uuid: "05d1046a70f84d86a215b76dc85ef109" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 39201 } } peers { permanent_uuid: "c270fa5b12c2489abe94f87b4539e7bc" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 44009 } }
I20260928 21:54:35.937974 10792 raft_consensus.cc:374] T 3542c748bdb34c55a8ef129bebeee9bf P 05d1046a70f84d86a215b76dc85ef109 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:54:35.938004 10792 raft_consensus.cc:729] T 3542c748bdb34c55a8ef129bebeee9bf P 05d1046a70f84d86a215b76dc85ef109 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05d1046a70f84d86a215b76dc85ef109, State: Initialized, Role: FOLLOWER
I20260928 21:54:35.938054 10792 consensus_queue.cc:260] T 3542c748bdb34c55a8ef129bebeee9bf P 05d1046a70f84d86a215b76dc85ef109 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c5ce1c5850d406fbd3f4eeca6fde124" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35485 } } peers { permanent_uuid: "05d1046a70f84d86a215b76dc85ef109" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 39201 } } peers { permanent_uuid: "c270fa5b12c2489abe94f87b4539e7bc" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 44009 } }
I20260928 21:54:35.938145 10792 ts_tablet_manager.cc:1428] T 3542c748bdb34c55a8ef129bebeee9bf P 05d1046a70f84d86a215b76dc85ef109: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 21:54:35.938189 10816 ts_tablet_manager.cc:1939] T 292d18401c544deeb0d5d814a64e4dab P 05d1046a70f84d86a215b76dc85ef109: Deleting consensus metadata
I20260928 21:54:35.937706 10817 ts_tablet_manager.cc:1918] T 292d18401c544deeb0d5d814a64e4dab P 8c5ce1c5850d406fbd3f4eeca6fde124: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:54:35.938488 10817 log.cc:1196] T 292d18401c544deeb0d5d814a64e4dab P 8c5ce1c5850d406fbd3f4eeca6fde124: Deleting WAL directory at /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/minicluster-data/ts-0/wal/wals/292d18401c544deeb0d5d814a64e4dab
I20260928 21:54:35.938694 10796 tablet_bootstrap.cc:492] T 3542c748bdb34c55a8ef129bebeee9bf P c270fa5b12c2489abe94f87b4539e7bc: No bootstrap required, opened a new log
I20260928 21:54:35.938741 10796 ts_tablet_manager.cc:1397] T 3542c748bdb34c55a8ef129bebeee9bf P c270fa5b12c2489abe94f87b4539e7bc: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260928 21:54:35.938820 10267 catalog_manager.cc:4841] TS 05d1046a70f84d86a215b76dc85ef109 (127.2.182.66:39201): tablet 292d18401c544deeb0d5d814a64e4dab (table test-table [id=2eec63287db64d0db25d725a884e44f0]) successfully deleted
I20260928 21:54:35.938876 10796 raft_consensus.cc:348] T 3542c748bdb34c55a8ef129bebeee9bf P c270fa5b12c2489abe94f87b4539e7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c5ce1c5850d406fbd3f4eeca6fde124" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35485 } } peers { permanent_uuid: "05d1046a70f84d86a215b76dc85ef109" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 39201 } } peers { permanent_uuid: "c270fa5b12c2489abe94f87b4539e7bc" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 44009 } }
I20260928 21:54:35.938969 10796 raft_consensus.cc:374] T 3542c748bdb34c55a8ef129bebeee9bf P c270fa5b12c2489abe94f87b4539e7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:54:35.938997 10796 raft_consensus.cc:729] T 3542c748bdb34c55a8ef129bebeee9bf P c270fa5b12c2489abe94f87b4539e7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c270fa5b12c2489abe94f87b4539e7bc, State: Initialized, Role: FOLLOWER
I20260928 21:54:35.939036 10796 consensus_queue.cc:260] T 3542c748bdb34c55a8ef129bebeee9bf P c270fa5b12c2489abe94f87b4539e7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c5ce1c5850d406fbd3f4eeca6fde124" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35485 } } peers { permanent_uuid: "05d1046a70f84d86a215b76dc85ef109" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 39201 } } peers { permanent_uuid: "c270fa5b12c2489abe94f87b4539e7bc" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 44009 } }
I20260928 21:54:35.939049 10817 ts_tablet_manager.cc:1939] T 292d18401c544deeb0d5d814a64e4dab P 8c5ce1c5850d406fbd3f4eeca6fde124: Deleting consensus metadata
I20260928 21:54:35.939121 10796 ts_tablet_manager.cc:1428] T 3542c748bdb34c55a8ef129bebeee9bf P c270fa5b12c2489abe94f87b4539e7bc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:35.939421 10266 catalog_manager.cc:4841] TS 8c5ce1c5850d406fbd3f4eeca6fde124 (127.2.182.65:35485): tablet 292d18401c544deeb0d5d814a64e4dab (table test-table [id=2eec63287db64d0db25d725a884e44f0]) successfully deleted
W20260928 21:54:36.004359 10485 tablet.cc:2369] T 3542c748bdb34c55a8ef129bebeee9bf P 8c5ce1c5850d406fbd3f4eeca6fde124: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 21:54:36.086669 10765 tablet.cc:2369] T 3542c748bdb34c55a8ef129bebeee9bf P c270fa5b12c2489abe94f87b4539e7bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:54:36.154721 10797 raft_consensus.cc:482] T 3542c748bdb34c55a8ef129bebeee9bf P 8c5ce1c5850d406fbd3f4eeca6fde124 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:54:36.154830 10797 raft_consensus.cc:504] T 3542c748bdb34c55a8ef129bebeee9bf P 8c5ce1c5850d406fbd3f4eeca6fde124 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c5ce1c5850d406fbd3f4eeca6fde124" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35485 } } peers { permanent_uuid: "05d1046a70f84d86a215b76dc85ef109" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 39201 } } peers { permanent_uuid: "c270fa5b12c2489abe94f87b4539e7bc" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 44009 } }
I20260928 21:54:36.155113 10797 leader_election.cc:288] T 3542c748bdb34c55a8ef129bebeee9bf P 8c5ce1c5850d406fbd3f4eeca6fde124 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05d1046a70f84d86a215b76dc85ef109 (127.2.182.66:39201), c270fa5b12c2489abe94f87b4539e7bc (127.2.182.67:44009)
I20260928 21:54:36.159430 10789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:36.155470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.65:58379 (local address 127.2.182.66:39201)
0928 21:54:36.155509 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:36.155511 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:36.155542 (+ 31us) server_negotiation.cc:407] Connection header received
0928 21:54:36.155624 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:36.155625 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:36.155635 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:36.155697 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:54:36.155980 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:36.156699 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:36.158264 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:36.158827 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:36.158915 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:36.158925 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:36.159238 (+ 313us) server_negotiation.cc:299] Negotiation successful
0928 21:54:36.159300 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 21:54:36.159399 10788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:36.155437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.65:55147 (local address 127.2.182.67:44009)
0928 21:54:36.155484 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:36.155486 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:36.155556 (+ 70us) server_negotiation.cc:407] Connection header received
0928 21:54:36.155638 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:36.155639 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:36.155650 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:36.155697 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:54:36.155934 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:36.156690 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:36.158272 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:36.158971 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:36.159023 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:36.159029 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:36.159248 (+ 219us) server_negotiation.cc:299] Negotiation successful
0928 21:54:36.159294 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 21:54:36.159390 10821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:36.155357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.67:44009 (local address 127.2.182.65:55147)
0928 21:54:36.155481 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:36.155493 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:36.155558 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:36.155710 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:36.155710 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:36.155713 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:54:36.155806 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:36.155809 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:36.156810 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:36.156811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:36.158126 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:36.158129 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:36.159050 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:36.159051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:36.159145 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:36.159147 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 21:54:36.159197 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 21:54:36.159238 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":36,"thread_start_us":64,"threads_started":1}
I20260928 21:54:36.159397 10822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:36.155361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.66:39201 (local address 127.2.182.65:58379)
0928 21:54:36.155481 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:36.155493 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:36.155558 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:36.155710 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:36.155710 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:36.155714 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:54:36.155905 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:36.155907 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:36.156873 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:36.156874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:36.158126 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:36.158129 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:36.158994 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:36.158995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:36.159109 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:36.159116 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 21:54:36.159184 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 21:54:36.159233 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":24,"thread_start_us":61,"threads_started":1}
I20260928 21:54:36.159758 10576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3542c748bdb34c55a8ef129bebeee9bf" candidate_uuid: "8c5ce1c5850d406fbd3f4eeca6fde124" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05d1046a70f84d86a215b76dc85ef109" is_pre_election: true
I20260928 21:54:36.159762 10702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3542c748bdb34c55a8ef129bebeee9bf" candidate_uuid: "8c5ce1c5850d406fbd3f4eeca6fde124" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c270fa5b12c2489abe94f87b4539e7bc" is_pre_election: true
I20260928 21:54:36.159847 10576 raft_consensus.cc:2454] T 3542c748bdb34c55a8ef129bebeee9bf P 05d1046a70f84d86a215b76dc85ef109 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c5ce1c5850d406fbd3f4eeca6fde124 in term 0.
I20260928 21:54:36.159927 10702 raft_consensus.cc:2454] T 3542c748bdb34c55a8ef129bebeee9bf P c270fa5b12c2489abe94f87b4539e7bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c5ce1c5850d406fbd3f4eeca6fde124 in term 0.
I20260928 21:54:36.160055 10371 leader_election.cc:302] T 3542c748bdb34c55a8ef129bebeee9bf P 8c5ce1c5850d406fbd3f4eeca6fde124 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05d1046a70f84d86a215b76dc85ef109, 8c5ce1c5850d406fbd3f4eeca6fde124; no voters:
I20260928 21:54:36.160241 10797 raft_consensus.cc:2781] T 3542c748bdb34c55a8ef129bebeee9bf P 8c5ce1c5850d406fbd3f4eeca6fde124 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:54:36.160298 10797 raft_consensus.cc:482] T 3542c748bdb34c55a8ef129bebeee9bf P 8c5ce1c5850d406fbd3f4eeca6fde124 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:54:36.160324 10797 raft_consensus.cc:3037] T 3542c748bdb34c55a8ef129bebeee9bf P 8c5ce1c5850d406fbd3f4eeca6fde124 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:54:36.161149 10797 raft_consensus.cc:504] T 3542c748bdb34c55a8ef129bebeee9bf P 8c5ce1c5850d406fbd3f4eeca6fde124 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c5ce1c5850d406fbd3f4eeca6fde124" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35485 } } peers { permanent_uuid: "05d1046a70f84d86a215b76dc85ef109" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 39201 } } peers { permanent_uuid: "c270fa5b12c2489abe94f87b4539e7bc" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 44009 } }
I20260928 21:54:36.161295 10797 leader_election.cc:288] T 3542c748bdb34c55a8ef129bebeee9bf P 8c5ce1c5850d406fbd3f4eeca6fde124 [CANDIDATE]: Term 1 election: Requested vote from peers 05d1046a70f84d86a215b76dc85ef109 (127.2.182.66:39201), c270fa5b12c2489abe94f87b4539e7bc (127.2.182.67:44009)
I20260928 21:54:36.161422 10576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3542c748bdb34c55a8ef129bebeee9bf" candidate_uuid: "8c5ce1c5850d406fbd3f4eeca6fde124" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05d1046a70f84d86a215b76dc85ef109"
I20260928 21:54:36.161509 10576 raft_consensus.cc:3037] T 3542c748bdb34c55a8ef129bebeee9bf P 05d1046a70f84d86a215b76dc85ef109 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:54:36.161492 10702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3542c748bdb34c55a8ef129bebeee9bf" candidate_uuid: "8c5ce1c5850d406fbd3f4eeca6fde124" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c270fa5b12c2489abe94f87b4539e7bc"
I20260928 21:54:36.161561 10702 raft_consensus.cc:3037] T 3542c748bdb34c55a8ef129bebeee9bf P c270fa5b12c2489abe94f87b4539e7bc [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:54:36.162160 10576 raft_consensus.cc:2454] T 3542c748bdb34c55a8ef129bebeee9bf P 05d1046a70f84d86a215b76dc85ef109 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c5ce1c5850d406fbd3f4eeca6fde124 in term 1.
I20260928 21:54:36.162209 10702 raft_consensus.cc:2454] T 3542c748bdb34c55a8ef129bebeee9bf P c270fa5b12c2489abe94f87b4539e7bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c5ce1c5850d406fbd3f4eeca6fde124 in term 1.
I20260928 21:54:36.162338 10371 leader_election.cc:302] T 3542c748bdb34c55a8ef129bebeee9bf P 8c5ce1c5850d406fbd3f4eeca6fde124 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05d1046a70f84d86a215b76dc85ef109, 8c5ce1c5850d406fbd3f4eeca6fde124; no voters:
I20260928 21:54:36.162456 10797 raft_consensus.cc:2781] T 3542c748bdb34c55a8ef129bebeee9bf P 8c5ce1c5850d406fbd3f4eeca6fde124 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:54:36.162566 10797 raft_consensus.cc:686] T 3542c748bdb34c55a8ef129bebeee9bf P 8c5ce1c5850d406fbd3f4eeca6fde124 [term 1 LEADER]: Becoming Leader. State: Replica: 8c5ce1c5850d406fbd3f4eeca6fde124, State: Running, Role: LEADER
I20260928 21:54:36.162631 10797 consensus_queue.cc:237] T 3542c748bdb34c55a8ef129bebeee9bf P 8c5ce1c5850d406fbd3f4eeca6fde124 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c5ce1c5850d406fbd3f4eeca6fde124" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35485 } } peers { permanent_uuid: "05d1046a70f84d86a215b76dc85ef109" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 39201 } } peers { permanent_uuid: "c270fa5b12c2489abe94f87b4539e7bc" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 44009 } }
I20260928 21:54:36.163286 10281 catalog_manager.cc:5495] T 3542c748bdb34c55a8ef129bebeee9bf P 8c5ce1c5850d406fbd3f4eeca6fde124 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c5ce1c5850d406fbd3f4eeca6fde124 (127.2.182.65). New cstate: current_term: 1 leader_uuid: "8c5ce1c5850d406fbd3f4eeca6fde124" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c5ce1c5850d406fbd3f4eeca6fde124" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05d1046a70f84d86a215b76dc85ef109" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 39201 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c270fa5b12c2489abe94f87b4539e7bc" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 44009 } health_report { overall_health: UNKNOWN } } }
Sep 28 21:54:36 dist-test-slave-k6lz krb5kdc[10236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632475, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.126@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM
I20260928 21:54:36.239058 10330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:36.232251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43124 (local address 127.2.182.126:45217)
0928 21:54:36.232315 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:36.232317 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:36.232348 (+ 31us) server_negotiation.cc:407] Connection header received
0928 21:54:36.232397 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:36.232398 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:36.232407 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:36.232456 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:36.233121 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:36.233626 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:36.234387 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:36.234577 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:36.234632 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:36.236758 (+ 2126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:36.236760 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:36.236760 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:36.236768 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:36.237988 (+ 1220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:36.238336 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:36.238338 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:36.238338 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:36.238385 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:36.238542 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:36.238543 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:36.238543 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:36.238644 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:36.238696 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:36.238839 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 21:54:36.238980 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":72,"server-negotiator.queue_time_us":40}
I20260928 21:54:36.239058 10830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:36.232114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:45217 (local address 127.0.0.1:43124)
0928 21:54:36.232275 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:36.232286 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:36.232345 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:36.232548 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:36.232550 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:36.232970 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:36.233021 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:36.233024 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:36.233715 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:36.233716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:36.234302 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:36.234304 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:36.234646 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:36.234647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:36.234691 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:36.235237 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:36.235241 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:36.236671 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:36.238063 (+ 1392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:36.238070 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:36.238073 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:36.238267 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:36.238439 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:36.238439 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:36.238440 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:36.238494 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:36.238698 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:36.238705 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:36.238750 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 21:54:36.238805 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 21:54:36.238930 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":63,"threads_started":1}
I20260928 21:54:36.241631 10281 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.2.182.126@KRBTEST.COM'} at 127.0.0.1:43124:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 21:54:36.241873 10281 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:54:36.243362 10558 tablet_service.cc:1467] Processing CreateTablet for tablet d912c40a5b074a93b14d76d5fa9b9583 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=96074a43c63e45a2bdf45775987b23e0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 21:54:36.243362 10698 tablet_service.cc:1467] Processing CreateTablet for tablet d912c40a5b074a93b14d76d5fa9b9583 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=96074a43c63e45a2bdf45775987b23e0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 21:54:36.243525 10698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d912c40a5b074a93b14d76d5fa9b9583. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:54:36.243525 10558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d912c40a5b074a93b14d76d5fa9b9583. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:54:36.243476 10418 tablet_service.cc:1467] Processing CreateTablet for tablet d912c40a5b074a93b14d76d5fa9b9583 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=96074a43c63e45a2bdf45775987b23e0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 21:54:36.243620 10418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d912c40a5b074a93b14d76d5fa9b9583. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:54:36.244803 10793 tablet_bootstrap.cc:492] T d912c40a5b074a93b14d76d5fa9b9583 P 8c5ce1c5850d406fbd3f4eeca6fde124: Bootstrap starting.
I20260928 21:54:36.244992 10796 tablet_bootstrap.cc:492] T d912c40a5b074a93b14d76d5fa9b9583 P c270fa5b12c2489abe94f87b4539e7bc: Bootstrap starting.
I20260928 21:54:36.245261 10793 tablet_bootstrap.cc:654] T d912c40a5b074a93b14d76d5fa9b9583 P 8c5ce1c5850d406fbd3f4eeca6fde124: Neither blocks nor log segments found. Creating new log.
I20260928 21:54:36.245260 10796 tablet_bootstrap.cc:654] T d912c40a5b074a93b14d76d5fa9b9583 P c270fa5b12c2489abe94f87b4539e7bc: Neither blocks nor log segments found. Creating new log.
I20260928 21:54:36.245517 10792 tablet_bootstrap.cc:492] T d912c40a5b074a93b14d76d5fa9b9583 P 05d1046a70f84d86a215b76dc85ef109: Bootstrap starting.
I20260928 21:54:36.245886 10793 tablet_bootstrap.cc:492] T d912c40a5b074a93b14d76d5fa9b9583 P 8c5ce1c5850d406fbd3f4eeca6fde124: No bootstrap required, opened a new log
I20260928 21:54:36.245904 10792 tablet_bootstrap.cc:654] T d912c40a5b074a93b14d76d5fa9b9583 P 05d1046a70f84d86a215b76dc85ef109: Neither blocks nor log segments found. Creating new log.
I20260928 21:54:36.245944 10793 ts_tablet_manager.cc:1397] T d912c40a5b074a93b14d76d5fa9b9583 P 8c5ce1c5850d406fbd3f4eeca6fde124: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:54:36.246156 10793 raft_consensus.cc:348] T d912c40a5b074a93b14d76d5fa9b9583 P 8c5ce1c5850d406fbd3f4eeca6fde124 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c5ce1c5850d406fbd3f4eeca6fde124" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35485 } } peers { permanent_uuid: "05d1046a70f84d86a215b76dc85ef109" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 39201 } } peers { permanent_uuid: "c270fa5b12c2489abe94f87b4539e7bc" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 44009 } }
I20260928 21:54:36.246289 10793 raft_consensus.cc:374] T d912c40a5b074a93b14d76d5fa9b9583 P 8c5ce1c5850d406fbd3f4eeca6fde124 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:54:36.246313 10793 raft_consensus.cc:729] T d912c40a5b074a93b14d76d5fa9b9583 P 8c5ce1c5850d406fbd3f4eeca6fde124 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c5ce1c5850d406fbd3f4eeca6fde124, State: Initialized, Role: FOLLOWER
I20260928 21:54:36.246344 10796 tablet_bootstrap.cc:492] T d912c40a5b074a93b14d76d5fa9b9583 P c270fa5b12c2489abe94f87b4539e7bc: No bootstrap required, opened a new log
I20260928 21:54:36.246362 10793 consensus_queue.cc:260] T d912c40a5b074a93b14d76d5fa9b9583 P 8c5ce1c5850d406fbd3f4eeca6fde124 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c5ce1c5850d406fbd3f4eeca6fde124" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35485 } } peers { permanent_uuid: "05d1046a70f84d86a215b76dc85ef109" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 39201 } } peers { permanent_uuid: "c270fa5b12c2489abe94f87b4539e7bc" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 44009 } }
I20260928 21:54:36.246382 10796 ts_tablet_manager.cc:1397] T d912c40a5b074a93b14d76d5fa9b9583 P c270fa5b12c2489abe94f87b4539e7bc: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:54:36.246469 10793 ts_tablet_manager.cc:1428] T d912c40a5b074a93b14d76d5fa9b9583 P 8c5ce1c5850d406fbd3f4eeca6fde124: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:36.246487 10796 raft_consensus.cc:348] T d912c40a5b074a93b14d76d5fa9b9583 P c270fa5b12c2489abe94f87b4539e7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c5ce1c5850d406fbd3f4eeca6fde124" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35485 } } peers { permanent_uuid: "05d1046a70f84d86a215b76dc85ef109" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 39201 } } peers { permanent_uuid: "c270fa5b12c2489abe94f87b4539e7bc" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 44009 } }
I20260928 21:54:36.246552 10796 raft_consensus.cc:374] T d912c40a5b074a93b14d76d5fa9b9583 P c270fa5b12c2489abe94f87b4539e7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:54:36.246577 10796 raft_consensus.cc:729] T d912c40a5b074a93b14d76d5fa9b9583 P c270fa5b12c2489abe94f87b4539e7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c270fa5b12c2489abe94f87b4539e7bc, State: Initialized, Role: FOLLOWER
I20260928 21:54:36.246519 10797 tablet_replica.cc:439] T d912c40a5b074a93b14d76d5fa9b9583 P 8c5ce1c5850d406fbd3f4eeca6fde124: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c5ce1c5850d406fbd3f4eeca6fde124" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35485 } } peers { permanent_uuid: "05d1046a70f84d86a215b76dc85ef109" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 39201 } } peers { permanent_uuid: "c270fa5b12c2489abe94f87b4539e7bc" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 44009 } } }
I20260928 21:54:36.246619 10796 consensus_queue.cc:260] T d912c40a5b074a93b14d76d5fa9b9583 P c270fa5b12c2489abe94f87b4539e7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c5ce1c5850d406fbd3f4eeca6fde124" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35485 } } peers { permanent_uuid: "05d1046a70f84d86a215b76dc85ef109" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 39201 } } peers { permanent_uuid: "c270fa5b12c2489abe94f87b4539e7bc" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 44009 } }
I20260928 21:54:36.246624 10797 tablet_replica.cc:442] T d912c40a5b074a93b14d76d5fa9b9583 P 8c5ce1c5850d406fbd3f4eeca6fde124: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:54:36.246706 10796 ts_tablet_manager.cc:1428] T d912c40a5b074a93b14d76d5fa9b9583 P c270fa5b12c2489abe94f87b4539e7bc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:36.246718 10800 tablet_replica.cc:439] T d912c40a5b074a93b14d76d5fa9b9583 P c270fa5b12c2489abe94f87b4539e7bc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c5ce1c5850d406fbd3f4eeca6fde124" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35485 } } peers { permanent_uuid: "05d1046a70f84d86a215b76dc85ef109" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 39201 } } peers { permanent_uuid: "c270fa5b12c2489abe94f87b4539e7bc" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 44009 } } }
I20260928 21:54:36.246804 10800 tablet_replica.cc:442] T d912c40a5b074a93b14d76d5fa9b9583 P c270fa5b12c2489abe94f87b4539e7bc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:54:36.246992 10792 tablet_bootstrap.cc:492] T d912c40a5b074a93b14d76d5fa9b9583 P 05d1046a70f84d86a215b76dc85ef109: No bootstrap required, opened a new log
I20260928 21:54:36.247032 10792 ts_tablet_manager.cc:1397] T d912c40a5b074a93b14d76d5fa9b9583 P 05d1046a70f84d86a215b76dc85ef109: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:54:36.247167 10792 raft_consensus.cc:348] T d912c40a5b074a93b14d76d5fa9b9583 P 05d1046a70f84d86a215b76dc85ef109 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c5ce1c5850d406fbd3f4eeca6fde124" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35485 } } peers { permanent_uuid: "05d1046a70f84d86a215b76dc85ef109" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 39201 } } peers { permanent_uuid: "c270fa5b12c2489abe94f87b4539e7bc" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 44009 } }
I20260928 21:54:36.247243 10792 raft_consensus.cc:374] T d912c40a5b074a93b14d76d5fa9b9583 P 05d1046a70f84d86a215b76dc85ef109 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:54:36.247269 10792 raft_consensus.cc:729] T d912c40a5b074a93b14d76d5fa9b9583 P 05d1046a70f84d86a215b76dc85ef109 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05d1046a70f84d86a215b76dc85ef109, State: Initialized, Role: FOLLOWER
I20260928 21:54:36.247308 10792 consensus_queue.cc:260] T d912c40a5b074a93b14d76d5fa9b9583 P 05d1046a70f84d86a215b76dc85ef109 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c5ce1c5850d406fbd3f4eeca6fde124" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35485 } } peers { permanent_uuid: "05d1046a70f84d86a215b76dc85ef109" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 39201 } } peers { permanent_uuid: "c270fa5b12c2489abe94f87b4539e7bc" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 44009 } }
I20260928 21:54:36.247426 10792 ts_tablet_manager.cc:1428] T d912c40a5b074a93b14d76d5fa9b9583 P 05d1046a70f84d86a215b76dc85ef109: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 21:54:36.247491 10798 tablet_replica.cc:439] T d912c40a5b074a93b14d76d5fa9b9583 P 05d1046a70f84d86a215b76dc85ef109: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c5ce1c5850d406fbd3f4eeca6fde124" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35485 } } peers { permanent_uuid: "05d1046a70f84d86a215b76dc85ef109" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 39201 } } peers { permanent_uuid: "c270fa5b12c2489abe94f87b4539e7bc" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 44009 } } }
I20260928 21:54:36.247558 10798 tablet_replica.cc:442] T d912c40a5b074a93b14d76d5fa9b9583 P 05d1046a70f84d86a215b76dc85ef109: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:54:36.313344 10798 raft_consensus.cc:482] T d912c40a5b074a93b14d76d5fa9b9583 P 05d1046a70f84d86a215b76dc85ef109 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:54:36.313452 10798 raft_consensus.cc:504] T d912c40a5b074a93b14d76d5fa9b9583 P 05d1046a70f84d86a215b76dc85ef109 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c5ce1c5850d406fbd3f4eeca6fde124" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35485 } } peers { permanent_uuid: "05d1046a70f84d86a215b76dc85ef109" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 39201 } } peers { permanent_uuid: "c270fa5b12c2489abe94f87b4539e7bc" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 44009 } }
I20260928 21:54:36.313759 10798 leader_election.cc:288] T d912c40a5b074a93b14d76d5fa9b9583 P 05d1046a70f84d86a215b76dc85ef109 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c5ce1c5850d406fbd3f4eeca6fde124 (127.2.182.65:35485), c270fa5b12c2489abe94f87b4539e7bc (127.2.182.67:44009)
I20260928 21:54:36.316803 10788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:36.314253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.66:49077 (local address 127.2.182.67:44009)
0928 21:54:36.314302 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:36.314305 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:36.314314 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:54:36.314426 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:36.314427 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:36.314436 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:36.314479 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:54:36.314642 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:36.315100 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:36.316124 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:36.316500 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:36.316560 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:36.316570 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:36.316690 (+ 120us) server_negotiation.cc:299] Negotiation successful
0928 21:54:36.316729 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 21:54:36.316807 10791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:36.314079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.66:57617 (local address 127.2.182.65:35485)
0928 21:54:36.314165 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:36.314168 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:36.314242 (+ 74us) server_negotiation.cc:407] Connection header received
0928 21:54:36.314375 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:36.314376 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:36.314385 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:36.314431 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:54:36.314585 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:36.315089 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:36.316124 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:36.316498 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:36.316560 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:36.316569 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:36.316699 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 21:54:36.316737 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 21:54:36.316840 10835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:36.314166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.67:44009 (local address 127.2.182.66:49077)
0928 21:54:36.314256 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:36.314266 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:36.314325 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:36.314496 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:36.314497 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:36.314499 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:54:36.314572 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:36.314573 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:36.315174 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:36.315175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:36.316037 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:36.316040 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:36.316573 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:36.316574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:36.316631 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:36.316635 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:54:36.316678 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 21:54:36.316722 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69,"mutex_wait_us":32,"thread_start_us":50,"threads_started":1}
I20260928 21:54:36.316946 10834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:36.313985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.65:35485 (local address 127.2.182.66:57617)
0928 21:54:36.314154 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:36.314168 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:36.314275 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:36.314448 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:36.314448 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:36.314450 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:54:36.314524 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:36.314526 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:36.315170 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:36.315172 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:36.316032 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:36.316034 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:36.316569 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:36.316570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:36.316638 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:36.316641 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:54:36.316685 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 21:54:36.316731 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":35,"thread_start_us":77,"threads_started":1}
I20260928 21:54:36.317277 10438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d912c40a5b074a93b14d76d5fa9b9583" candidate_uuid: "05d1046a70f84d86a215b76dc85ef109" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c5ce1c5850d406fbd3f4eeca6fde124" is_pre_election: true
I20260928 21:54:36.317368 10438 raft_consensus.cc:2454] T d912c40a5b074a93b14d76d5fa9b9583 P 8c5ce1c5850d406fbd3f4eeca6fde124 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05d1046a70f84d86a215b76dc85ef109 in term 0.
I20260928 21:54:36.317471 10702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d912c40a5b074a93b14d76d5fa9b9583" candidate_uuid: "05d1046a70f84d86a215b76dc85ef109" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c270fa5b12c2489abe94f87b4539e7bc" is_pre_election: true
I20260928 21:54:36.317552 10702 raft_consensus.cc:2454] T d912c40a5b074a93b14d76d5fa9b9583 P c270fa5b12c2489abe94f87b4539e7bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05d1046a70f84d86a215b76dc85ef109 in term 0.
I20260928 21:54:36.317587 10510 leader_election.cc:302] T d912c40a5b074a93b14d76d5fa9b9583 P 05d1046a70f84d86a215b76dc85ef109 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05d1046a70f84d86a215b76dc85ef109, 8c5ce1c5850d406fbd3f4eeca6fde124; no voters:
I20260928 21:54:36.317759 10798 raft_consensus.cc:2781] T d912c40a5b074a93b14d76d5fa9b9583 P 05d1046a70f84d86a215b76dc85ef109 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:54:36.317807 10798 raft_consensus.cc:482] T d912c40a5b074a93b14d76d5fa9b9583 P 05d1046a70f84d86a215b76dc85ef109 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:54:36.317822 10798 raft_consensus.cc:3037] T d912c40a5b074a93b14d76d5fa9b9583 P 05d1046a70f84d86a215b76dc85ef109 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:54:36.318668 10798 raft_consensus.cc:504] T d912c40a5b074a93b14d76d5fa9b9583 P 05d1046a70f84d86a215b76dc85ef109 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c5ce1c5850d406fbd3f4eeca6fde124" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35485 } } peers { permanent_uuid: "05d1046a70f84d86a215b76dc85ef109" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 39201 } } peers { permanent_uuid: "c270fa5b12c2489abe94f87b4539e7bc" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 44009 } }
I20260928 21:54:36.318789 10798 leader_election.cc:288] T d912c40a5b074a93b14d76d5fa9b9583 P 05d1046a70f84d86a215b76dc85ef109 [CANDIDATE]: Term 1 election: Requested vote from peers 8c5ce1c5850d406fbd3f4eeca6fde124 (127.2.182.65:35485), c270fa5b12c2489abe94f87b4539e7bc (127.2.182.67:44009)
I20260928 21:54:36.319058 10702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d912c40a5b074a93b14d76d5fa9b9583" candidate_uuid: "05d1046a70f84d86a215b76dc85ef109" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c270fa5b12c2489abe94f87b4539e7bc"
I20260928 21:54:36.319051 10438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d912c40a5b074a93b14d76d5fa9b9583" candidate_uuid: "05d1046a70f84d86a215b76dc85ef109" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c5ce1c5850d406fbd3f4eeca6fde124"
I20260928 21:54:36.319151 10702 raft_consensus.cc:3037] T d912c40a5b074a93b14d76d5fa9b9583 P c270fa5b12c2489abe94f87b4539e7bc [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:54:36.319151 10438 raft_consensus.cc:3037] T d912c40a5b074a93b14d76d5fa9b9583 P 8c5ce1c5850d406fbd3f4eeca6fde124 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:54:36.319960 10702 raft_consensus.cc:2454] T d912c40a5b074a93b14d76d5fa9b9583 P c270fa5b12c2489abe94f87b4539e7bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05d1046a70f84d86a215b76dc85ef109 in term 1.
I20260928 21:54:36.320036 10438 raft_consensus.cc:2454] T d912c40a5b074a93b14d76d5fa9b9583 P 8c5ce1c5850d406fbd3f4eeca6fde124 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05d1046a70f84d86a215b76dc85ef109 in term 1.
I20260928 21:54:36.320154 10510 leader_election.cc:302] T d912c40a5b074a93b14d76d5fa9b9583 P 05d1046a70f84d86a215b76dc85ef109 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05d1046a70f84d86a215b76dc85ef109, 8c5ce1c5850d406fbd3f4eeca6fde124; no voters:
I20260928 21:54:36.320245 10798 raft_consensus.cc:2781] T d912c40a5b074a93b14d76d5fa9b9583 P 05d1046a70f84d86a215b76dc85ef109 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:54:36.320393 10798 raft_consensus.cc:686] T d912c40a5b074a93b14d76d5fa9b9583 P 05d1046a70f84d86a215b76dc85ef109 [term 1 LEADER]: Becoming Leader. State: Replica: 05d1046a70f84d86a215b76dc85ef109, State: Running, Role: LEADER
I20260928 21:54:36.320461 10798 consensus_queue.cc:237] T d912c40a5b074a93b14d76d5fa9b9583 P 05d1046a70f84d86a215b76dc85ef109 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c5ce1c5850d406fbd3f4eeca6fde124" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35485 } } peers { permanent_uuid: "05d1046a70f84d86a215b76dc85ef109" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 39201 } } peers { permanent_uuid: "c270fa5b12c2489abe94f87b4539e7bc" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 44009 } }
I20260928 21:54:36.320820 10836 tablet_replica.cc:439] T d912c40a5b074a93b14d76d5fa9b9583 P 05d1046a70f84d86a215b76dc85ef109: TxnStatusTablet state changed. Reason: New leader 05d1046a70f84d86a215b76dc85ef109. Latest consensus state: current_term: 1 leader_uuid: "05d1046a70f84d86a215b76dc85ef109" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c5ce1c5850d406fbd3f4eeca6fde124" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35485 } } peers { permanent_uuid: "05d1046a70f84d86a215b76dc85ef109" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 39201 } } peers { permanent_uuid: "c270fa5b12c2489abe94f87b4539e7bc" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 44009 } } }
I20260928 21:54:36.320924 10836 tablet_replica.cc:442] T d912c40a5b074a93b14d76d5fa9b9583 P 05d1046a70f84d86a215b76dc85ef109: This TxnStatusTablet replica's current role is: LEADER
I20260928 21:54:36.321096 10838 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 21:54:36.321202 10838 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 21:54:36.321215 10281 catalog_manager.cc:5495] T d912c40a5b074a93b14d76d5fa9b9583 P 05d1046a70f84d86a215b76dc85ef109 reported cstate change: term changed from 0 to 1, leader changed from <none> to 05d1046a70f84d86a215b76dc85ef109 (127.2.182.66). New cstate: current_term: 1 leader_uuid: "05d1046a70f84d86a215b76dc85ef109" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c5ce1c5850d406fbd3f4eeca6fde124" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05d1046a70f84d86a215b76dc85ef109" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 39201 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c270fa5b12c2489abe94f87b4539e7bc" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 44009 } health_report { overall_health: UNKNOWN } } }
I20260928 21:54:36.342319 10789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:36.339535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34260 (local address 127.2.182.66:39201)
0928 21:54:36.339577 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:36.339580 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:36.339635 (+ 55us) server_negotiation.cc:407] Connection header received
0928 21:54:36.339728 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:36.339729 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:36.339745 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:36.339799 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:54:36.339969 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:36.340453 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:36.341574 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:36.341763 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:36.341821 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:36.342013 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:54:36.342118 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:54:36.342166 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:36.342227 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 21:54:36.342242 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 21:54:36.342358 10830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:36.339510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.66:39201 (local address 127.0.0.1:34260)
0928 21:54:36.339593 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:36.339604 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:36.339653 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:36.339805 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:36.339806 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:36.339807 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 21:54:36.339864 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:36.339874 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:36.340538 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:36.340539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:36.341481 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:36.341484 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:36.341831 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:36.341832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:36.341907 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:36.341928 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 21:54:36.342168 (+ 240us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 21:54:36.342174 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 21:54:36.342223 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 21:54:36.342271 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 21:54:36.343559 10438 raft_consensus.cc:1261] T d912c40a5b074a93b14d76d5fa9b9583 P 8c5ce1c5850d406fbd3f4eeca6fde124 [term 1 FOLLOWER]: Refusing update from remote peer 05d1046a70f84d86a215b76dc85ef109: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:54:36.343627 10702 raft_consensus.cc:1261] T d912c40a5b074a93b14d76d5fa9b9583 P c270fa5b12c2489abe94f87b4539e7bc [term 1 FOLLOWER]: Refusing update from remote peer 05d1046a70f84d86a215b76dc85ef109: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:54:36.344018 10798 consensus_queue.cc:1035] T d912c40a5b074a93b14d76d5fa9b9583 P 05d1046a70f84d86a215b76dc85ef109 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c270fa5b12c2489abe94f87b4539e7bc" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 44009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:54:36.344221 10836 consensus_queue.cc:1035] T d912c40a5b074a93b14d76d5fa9b9583 P 05d1046a70f84d86a215b76dc85ef109 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c5ce1c5850d406fbd3f4eeca6fde124" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:54:36.344710 10800 tablet_replica.cc:439] T d912c40a5b074a93b14d76d5fa9b9583 P c270fa5b12c2489abe94f87b4539e7bc: TxnStatusTablet state changed. Reason: New leader 05d1046a70f84d86a215b76dc85ef109. Latest consensus state: current_term: 1 leader_uuid: "05d1046a70f84d86a215b76dc85ef109" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c5ce1c5850d406fbd3f4eeca6fde124" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35485 } } peers { permanent_uuid: "05d1046a70f84d86a215b76dc85ef109" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 39201 } } peers { permanent_uuid: "c270fa5b12c2489abe94f87b4539e7bc" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 44009 } } }
I20260928 21:54:36.344810 10800 tablet_replica.cc:442] T d912c40a5b074a93b14d76d5fa9b9583 P c270fa5b12c2489abe94f87b4539e7bc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:54:36.344750 10797 tablet_replica.cc:439] T d912c40a5b074a93b14d76d5fa9b9583 P 8c5ce1c5850d406fbd3f4eeca6fde124: TxnStatusTablet state changed. Reason: New leader 05d1046a70f84d86a215b76dc85ef109. Latest consensus state: current_term: 1 leader_uuid: "05d1046a70f84d86a215b76dc85ef109" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c5ce1c5850d406fbd3f4eeca6fde124" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35485 } } peers { permanent_uuid: "05d1046a70f84d86a215b76dc85ef109" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 39201 } } peers { permanent_uuid: "c270fa5b12c2489abe94f87b4539e7bc" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 44009 } } }
I20260928 21:54:36.344835 10797 tablet_replica.cc:442] T d912c40a5b074a93b14d76d5fa9b9583 P 8c5ce1c5850d406fbd3f4eeca6fde124: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:54:36.345322 10836 tablet_replica.cc:439] T d912c40a5b074a93b14d76d5fa9b9583 P 05d1046a70f84d86a215b76dc85ef109: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "05d1046a70f84d86a215b76dc85ef109" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c5ce1c5850d406fbd3f4eeca6fde124" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35485 } } peers { permanent_uuid: "05d1046a70f84d86a215b76dc85ef109" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 39201 } } peers { permanent_uuid: "c270fa5b12c2489abe94f87b4539e7bc" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 44009 } } }
I20260928 21:54:36.345407 10836 tablet_replica.cc:442] T d912c40a5b074a93b14d76d5fa9b9583 P 05d1046a70f84d86a215b76dc85ef109: This TxnStatusTablet replica's current role is: LEADER
I20260928 21:54:36.346453 10836 tablet_replica.cc:439] T d912c40a5b074a93b14d76d5fa9b9583 P 05d1046a70f84d86a215b76dc85ef109: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "05d1046a70f84d86a215b76dc85ef109" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c5ce1c5850d406fbd3f4eeca6fde124" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35485 } } peers { permanent_uuid: "05d1046a70f84d86a215b76dc85ef109" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 39201 } } peers { permanent_uuid: "c270fa5b12c2489abe94f87b4539e7bc" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 44009 } } }
I20260928 21:54:36.346534 10836 tablet_replica.cc:442] T d912c40a5b074a93b14d76d5fa9b9583 P 05d1046a70f84d86a215b76dc85ef109: This TxnStatusTablet replica's current role is: LEADER
I20260928 21:54:36.346642 10797 tablet_replica.cc:439] T d912c40a5b074a93b14d76d5fa9b9583 P 8c5ce1c5850d406fbd3f4eeca6fde124: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "05d1046a70f84d86a215b76dc85ef109" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c5ce1c5850d406fbd3f4eeca6fde124" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35485 } } peers { permanent_uuid: "05d1046a70f84d86a215b76dc85ef109" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 39201 } } peers { permanent_uuid: "c270fa5b12c2489abe94f87b4539e7bc" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 44009 } } }
I20260928 21:54:36.346717 10797 tablet_replica.cc:442] T d912c40a5b074a93b14d76d5fa9b9583 P 8c5ce1c5850d406fbd3f4eeca6fde124: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:54:36.346727 10800 tablet_replica.cc:439] T d912c40a5b074a93b14d76d5fa9b9583 P c270fa5b12c2489abe94f87b4539e7bc: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "05d1046a70f84d86a215b76dc85ef109" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c5ce1c5850d406fbd3f4eeca6fde124" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35485 } } peers { permanent_uuid: "05d1046a70f84d86a215b76dc85ef109" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 39201 } } peers { permanent_uuid: "c270fa5b12c2489abe94f87b4539e7bc" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 44009 } } }
I20260928 21:54:36.346817 10800 tablet_replica.cc:442] T d912c40a5b074a93b14d76d5fa9b9583 P c270fa5b12c2489abe94f87b4539e7bc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:54:36.350119 10791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:36.347376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56528 (local address 127.2.182.65:35485)
0928 21:54:36.347532 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:36.347534 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:36.347541 (+ 7us) server_negotiation.cc:407] Connection header received
0928 21:54:36.347575 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:36.347576 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:36.347598 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:36.347659 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:54:36.347794 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:36.348529 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:36.349232 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:36.349531 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:36.349598 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:36.349756 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:54:36.349897 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:54:36.349967 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:36.350024 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 21:54:36.350047 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137}
I20260928 21:54:36.355115 10845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:36.352682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.66:39201 (local address 127.2.182.65:41183)
0928 21:54:36.352829 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:36.352840 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:36.352886 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:36.353156 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:36.353157 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:36.353158 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 21:54:36.353219 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:36.353221 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:36.353837 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:36.353838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:36.354418 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:36.354419 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:36.354719 (+ 300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:36.354720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:36.354795 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:36.354808 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 21:54:36.354948 (+ 140us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 21:54:36.354949 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 21:54:36.354984 (+ 35us) client_negotiation.cc:240] Negotiation successful
0928 21:54:36.355027 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":75,"threads_started":1}
I20260928 21:54:36.355094 10789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:36.352992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.65:41183 (local address 127.2.182.66:39201)
0928 21:54:36.353054 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:36.353060 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:36.353066 (+ 6us) server_negotiation.cc:407] Connection header received
0928 21:54:36.353088 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:36.353089 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:36.353097 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:36.353144 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:54:36.353280 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:36.353764 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:36.354486 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:36.354663 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:36.354716 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:36.354873 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:54:36.354902 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:54:36.354941 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:36.354996 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 21:54:36.355014 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 21:54:36.357801 10702 raft_consensus.cc:1261] T 3542c748bdb34c55a8ef129bebeee9bf P c270fa5b12c2489abe94f87b4539e7bc [term 1 FOLLOWER]: Refusing update from remote peer 8c5ce1c5850d406fbd3f4eeca6fde124: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:54:36.357825 10576 raft_consensus.cc:1261] T 3542c748bdb34c55a8ef129bebeee9bf P 05d1046a70f84d86a215b76dc85ef109 [term 1 FOLLOWER]: Refusing update from remote peer 8c5ce1c5850d406fbd3f4eeca6fde124: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:54:36.358062 10797 consensus_queue.cc:1035] T 3542c748bdb34c55a8ef129bebeee9bf P 8c5ce1c5850d406fbd3f4eeca6fde124 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c270fa5b12c2489abe94f87b4539e7bc" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 44009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:54:36.358342 10823 consensus_queue.cc:1035] T 3542c748bdb34c55a8ef129bebeee9bf P 8c5ce1c5850d406fbd3f4eeca6fde124 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05d1046a70f84d86a215b76dc85ef109" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 39201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:54:36.368415 10791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:36.365812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.66:52061 (local address 127.2.182.65:35485)
0928 21:54:36.365852 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:36.365854 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:36.365996 (+ 142us) server_negotiation.cc:407] Connection header received
0928 21:54:36.366046 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:36.366047 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:36.366058 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:36.366101 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:54:36.366242 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:36.366836 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:36.367778 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:36.367984 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:36.368038 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:36.368189 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:54:36.368223 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:54:36.368265 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:36.368315 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 21:54:36.368335 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 21:54:36.368443 10851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:36.365774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.65:35485 (local address 127.2.182.66:52061)
0928 21:54:36.365914 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:36.365926 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:36.365997 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:36.366102 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:36.366102 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:36.366104 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 21:54:36.366188 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:36.366190 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:36.366915 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:36.366916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:36.367707 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:36.367710 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:36.368043 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:36.368044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:36.368107 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:36.368123 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 21:54:36.368272 (+ 149us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 21:54:36.368274 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 21:54:36.368313 (+ 39us) client_negotiation.cc:240] Negotiation successful
0928 21:54:36.368353 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":75,"threads_started":1}
I20260928 21:54:36.377709 10281 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43112:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:54:36.377833 10281 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43112:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:54:36.378657 10281 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 85cac027afc849588f7eae9726ee1eb1: Sending DeleteTablet for 3 replicas of tablet 3542c748bdb34c55a8ef129bebeee9bf
I20260928 21:54:36.379016 10558 tablet_service.cc:1514] Processing DeleteTablet for tablet 3542c748bdb34c55a8ef129bebeee9bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:54:36 UTC) from {username='oryx', principal='oryx/127.2.182.126@KRBTEST.COM'} at 127.0.0.1:34258
I20260928 21:54:36.379043 10418 tablet_service.cc:1514] Processing DeleteTablet for tablet 3542c748bdb34c55a8ef129bebeee9bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:54:36 UTC) from {username='oryx', principal='oryx/127.2.182.126@KRBTEST.COM'} at 127.0.0.1:56518
I20260928 21:54:36.379120 10816 tablet_replica.cc:330] T 3542c748bdb34c55a8ef129bebeee9bf P 05d1046a70f84d86a215b76dc85ef109: stopping tablet replica
I20260928 21:54:36.379134 10698 tablet_service.cc:1514] Processing DeleteTablet for tablet 3542c748bdb34c55a8ef129bebeee9bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:54:36 UTC) from {username='oryx', principal='oryx/127.2.182.126@KRBTEST.COM'} at 127.0.0.1:40838
I20260928 21:54:36.379168 10816 raft_consensus.cc:2229] T 3542c748bdb34c55a8ef129bebeee9bf P 05d1046a70f84d86a215b76dc85ef109 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:54:36.379202 10816 raft_consensus.cc:2258] T 3542c748bdb34c55a8ef129bebeee9bf P 05d1046a70f84d86a215b76dc85ef109 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:54:36.379222 10817 tablet_replica.cc:330] T 3542c748bdb34c55a8ef129bebeee9bf P 8c5ce1c5850d406fbd3f4eeca6fde124: stopping tablet replica
I20260928 21:54:36.379271 10817 raft_consensus.cc:2229] T 3542c748bdb34c55a8ef129bebeee9bf P 8c5ce1c5850d406fbd3f4eeca6fde124 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:54:36.379393 10815 tablet_replica.cc:330] T 3542c748bdb34c55a8ef129bebeee9bf P c270fa5b12c2489abe94f87b4539e7bc: stopping tablet replica
I20260928 21:54:36.379446 10815 raft_consensus.cc:2229] T 3542c748bdb34c55a8ef129bebeee9bf P c270fa5b12c2489abe94f87b4539e7bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:54:36.379487 10815 raft_consensus.cc:2258] T 3542c748bdb34c55a8ef129bebeee9bf P c270fa5b12c2489abe94f87b4539e7bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:54:36.379936 10816 ts_tablet_manager.cc:1905] T 3542c748bdb34c55a8ef129bebeee9bf P 05d1046a70f84d86a215b76dc85ef109: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:54:36.379971 10817 raft_consensus.cc:2258] T 3542c748bdb34c55a8ef129bebeee9bf P 8c5ce1c5850d406fbd3f4eeca6fde124 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:54:36.380167 10815 ts_tablet_manager.cc:1905] T 3542c748bdb34c55a8ef129bebeee9bf P c270fa5b12c2489abe94f87b4539e7bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:54:36.381896 10816 ts_tablet_manager.cc:1918] T 3542c748bdb34c55a8ef129bebeee9bf P 05d1046a70f84d86a215b76dc85ef109: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:54:36.381915 10817 ts_tablet_manager.cc:1905] T 3542c748bdb34c55a8ef129bebeee9bf P 8c5ce1c5850d406fbd3f4eeca6fde124: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:54:36.381959 10816 log.cc:1196] T 3542c748bdb34c55a8ef129bebeee9bf P 05d1046a70f84d86a215b76dc85ef109: Deleting WAL directory at /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/minicluster-data/ts-1/wal/wals/3542c748bdb34c55a8ef129bebeee9bf
I20260928 21:54:36.382153 10816 ts_tablet_manager.cc:1939] T 3542c748bdb34c55a8ef129bebeee9bf P 05d1046a70f84d86a215b76dc85ef109: Deleting consensus metadata
I20260928 21:54:36.383567 10817 ts_tablet_manager.cc:1918] T 3542c748bdb34c55a8ef129bebeee9bf P 8c5ce1c5850d406fbd3f4eeca6fde124: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:54:36.383620 10817 log.cc:1196] T 3542c748bdb34c55a8ef129bebeee9bf P 8c5ce1c5850d406fbd3f4eeca6fde124: Deleting WAL directory at /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/minicluster-data/ts-0/wal/wals/3542c748bdb34c55a8ef129bebeee9bf
I20260928 21:54:36.383814 10817 ts_tablet_manager.cc:1939] T 3542c748bdb34c55a8ef129bebeee9bf P 8c5ce1c5850d406fbd3f4eeca6fde124: Deleting consensus metadata
I20260928 21:54:36.384167 10266 catalog_manager.cc:4841] TS 8c5ce1c5850d406fbd3f4eeca6fde124 (127.2.182.65:35485): tablet 3542c748bdb34c55a8ef129bebeee9bf (table test-table [id=a243e97f7adc49529e99154b5ea30298]) successfully deleted
I20260928 21:54:36.384248 10267 catalog_manager.cc:4841] TS 05d1046a70f84d86a215b76dc85ef109 (127.2.182.66:39201): tablet 3542c748bdb34c55a8ef129bebeee9bf (table test-table [id=a243e97f7adc49529e99154b5ea30298]) successfully deleted
I20260928 21:54:36.384904 10815 ts_tablet_manager.cc:1918] T 3542c748bdb34c55a8ef129bebeee9bf P c270fa5b12c2489abe94f87b4539e7bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:54:36.384975 10815 log.cc:1196] T 3542c748bdb34c55a8ef129bebeee9bf P c270fa5b12c2489abe94f87b4539e7bc: Deleting WAL directory at /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/minicluster-data/ts-2/wal/wals/3542c748bdb34c55a8ef129bebeee9bf
I20260928 21:54:36.385169 10815 ts_tablet_manager.cc:1939] T 3542c748bdb34c55a8ef129bebeee9bf P c270fa5b12c2489abe94f87b4539e7bc: Deleting consensus metadata
I20260928 21:54:36.385492 10266 catalog_manager.cc:4841] TS c270fa5b12c2489abe94f87b4539e7bc (127.2.182.67:44009): tablet 3542c748bdb34c55a8ef129bebeee9bf (table test-table [id=a243e97f7adc49529e99154b5ea30298]) successfully deleted
Sep 28 21:54:36 dist-test-slave-k6lz krb5kdc[10236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 21:54:36 dist-test-slave-k6lz krb5kdc[10236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:54:36 dist-test-slave-k6lz krb5kdc[10236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
W20260928 21:54:36.414335 10330 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:54:36.407267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43126 (local address 127.2.182.126:45217)
0928 21:54:36.407321 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:36.407324 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:36.407524 (+ 200us) server_negotiation.cc:407] Connection header received
0928 21:54:36.407584 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:36.407585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:36.407598 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:36.407675 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:36.408466 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:36.409291 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:36.410391 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:36.410676 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:36.410755 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:36.414264 (+ 3509us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43126: BlockingRecv error: recv got EOF from 127.0.0.1:43126 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 28 21:54:36 dist-test-slave-k6lz krb5kdc[10236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM
I20260928 21:54:36.424630 10330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:36.415853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43132 (local address 127.2.182.126:45217)
0928 21:54:36.415920 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:36.415923 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:36.415989 (+ 66us) server_negotiation.cc:407] Connection header received
0928 21:54:36.416061 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:36.416062 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:36.416073 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:36.416131 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:36.416699 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:36.417480 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:36.418451 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:36.418754 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:36.418824 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:36.421585 (+ 2761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:36.421587 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:36.421588 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:36.421596 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:36.423305 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:36.423734 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:36.423735 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:36.423736 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:36.423796 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:36.423999 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:36.424000 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:36.424001 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:36.424139 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:36.424217 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:36.424446 (+ 229us) server_negotiation.cc:299] Negotiation successful
0928 21:54:36.424530 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 21:54:36.426863 10281 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43132:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:54:36.427042 10281 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:54:36.428774 10418 tablet_service.cc:1467] Processing CreateTablet for tablet 56299f48ff1640659913dee3cb63787a (DEFAULT_TABLE table=test-table [id=af5fd92e7eff461499a517588f7f0020]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:54:36.428916 10418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56299f48ff1640659913dee3cb63787a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:54:36.428898 10698 tablet_service.cc:1467] Processing CreateTablet for tablet 56299f48ff1640659913dee3cb63787a (DEFAULT_TABLE table=test-table [id=af5fd92e7eff461499a517588f7f0020]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:54:36.429024 10698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56299f48ff1640659913dee3cb63787a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:54:36.430380 10793 tablet_bootstrap.cc:492] T 56299f48ff1640659913dee3cb63787a P 8c5ce1c5850d406fbd3f4eeca6fde124: Bootstrap starting.
I20260928 21:54:36.430380 10558 tablet_service.cc:1467] Processing CreateTablet for tablet 56299f48ff1640659913dee3cb63787a (DEFAULT_TABLE table=test-table [id=af5fd92e7eff461499a517588f7f0020]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:54:36.430505 10558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56299f48ff1640659913dee3cb63787a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:54:36.430802 10793 tablet_bootstrap.cc:654] T 56299f48ff1640659913dee3cb63787a P 8c5ce1c5850d406fbd3f4eeca6fde124: Neither blocks nor log segments found. Creating new log.
I20260928 21:54:36.431412 10796 tablet_bootstrap.cc:492] T 56299f48ff1640659913dee3cb63787a P c270fa5b12c2489abe94f87b4539e7bc: Bootstrap starting.
I20260928 21:54:36.431734 10796 tablet_bootstrap.cc:654] T 56299f48ff1640659913dee3cb63787a P c270fa5b12c2489abe94f87b4539e7bc: Neither blocks nor log segments found. Creating new log.
I20260928 21:54:36.432169 10793 tablet_bootstrap.cc:492] T 56299f48ff1640659913dee3cb63787a P 8c5ce1c5850d406fbd3f4eeca6fde124: No bootstrap required, opened a new log
I20260928 21:54:36.432216 10793 ts_tablet_manager.cc:1397] T 56299f48ff1640659913dee3cb63787a P 8c5ce1c5850d406fbd3f4eeca6fde124: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:54:36.432381 10793 raft_consensus.cc:348] T 56299f48ff1640659913dee3cb63787a P 8c5ce1c5850d406fbd3f4eeca6fde124 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c5ce1c5850d406fbd3f4eeca6fde124" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35485 } } peers { permanent_uuid: "05d1046a70f84d86a215b76dc85ef109" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 39201 } } peers { permanent_uuid: "c270fa5b12c2489abe94f87b4539e7bc" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 44009 } }
I20260928 21:54:36.432451 10793 raft_consensus.cc:374] T 56299f48ff1640659913dee3cb63787a P 8c5ce1c5850d406fbd3f4eeca6fde124 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:54:36.432468 10793 raft_consensus.cc:729] T 56299f48ff1640659913dee3cb63787a P 8c5ce1c5850d406fbd3f4eeca6fde124 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c5ce1c5850d406fbd3f4eeca6fde124, State: Initialized, Role: FOLLOWER
I20260928 21:54:36.432507 10793 consensus_queue.cc:260] T 56299f48ff1640659913dee3cb63787a P 8c5ce1c5850d406fbd3f4eeca6fde124 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c5ce1c5850d406fbd3f4eeca6fde124" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35485 } } peers { permanent_uuid: "05d1046a70f84d86a215b76dc85ef109" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 39201 } } peers { permanent_uuid: "c270fa5b12c2489abe94f87b4539e7bc" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 44009 } }
I20260928 21:54:36.432595 10793 ts_tablet_manager.cc:1428] T 56299f48ff1640659913dee3cb63787a P 8c5ce1c5850d406fbd3f4eeca6fde124: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:36.432765 10792 tablet_bootstrap.cc:492] T 56299f48ff1640659913dee3cb63787a P 05d1046a70f84d86a215b76dc85ef109: Bootstrap starting.
I20260928 21:54:36.433127 10792 tablet_bootstrap.cc:654] T 56299f48ff1640659913dee3cb63787a P 05d1046a70f84d86a215b76dc85ef109: Neither blocks nor log segments found. Creating new log.
I20260928 21:54:36.433162 10796 tablet_bootstrap.cc:492] T 56299f48ff1640659913dee3cb63787a P c270fa5b12c2489abe94f87b4539e7bc: No bootstrap required, opened a new log
I20260928 21:54:36.433209 10796 ts_tablet_manager.cc:1397] T 56299f48ff1640659913dee3cb63787a P c270fa5b12c2489abe94f87b4539e7bc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:54:36.433305 10796 raft_consensus.cc:348] T 56299f48ff1640659913dee3cb63787a P c270fa5b12c2489abe94f87b4539e7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c5ce1c5850d406fbd3f4eeca6fde124" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35485 } } peers { permanent_uuid: "05d1046a70f84d86a215b76dc85ef109" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 39201 } } peers { permanent_uuid: "c270fa5b12c2489abe94f87b4539e7bc" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 44009 } }
I20260928 21:54:36.433370 10796 raft_consensus.cc:374] T 56299f48ff1640659913dee3cb63787a P c270fa5b12c2489abe94f87b4539e7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:54:36.433408 10796 raft_consensus.cc:729] T 56299f48ff1640659913dee3cb63787a P c270fa5b12c2489abe94f87b4539e7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c270fa5b12c2489abe94f87b4539e7bc, State: Initialized, Role: FOLLOWER
I20260928 21:54:36.433445 10796 consensus_queue.cc:260] T 56299f48ff1640659913dee3cb63787a P c270fa5b12c2489abe94f87b4539e7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c5ce1c5850d406fbd3f4eeca6fde124" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35485 } } peers { permanent_uuid: "05d1046a70f84d86a215b76dc85ef109" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 39201 } } peers { permanent_uuid: "c270fa5b12c2489abe94f87b4539e7bc" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 44009 } }
I20260928 21:54:36.433526 10796 ts_tablet_manager.cc:1428] T 56299f48ff1640659913dee3cb63787a P c270fa5b12c2489abe94f87b4539e7bc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:36.435817 10792 tablet_bootstrap.cc:492] T 56299f48ff1640659913dee3cb63787a P 05d1046a70f84d86a215b76dc85ef109: No bootstrap required, opened a new log
I20260928 21:54:36.435878 10792 ts_tablet_manager.cc:1397] T 56299f48ff1640659913dee3cb63787a P 05d1046a70f84d86a215b76dc85ef109: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 21:54:36.436034 10792 raft_consensus.cc:348] T 56299f48ff1640659913dee3cb63787a P 05d1046a70f84d86a215b76dc85ef109 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c5ce1c5850d406fbd3f4eeca6fde124" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35485 } } peers { permanent_uuid: "05d1046a70f84d86a215b76dc85ef109" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 39201 } } peers { permanent_uuid: "c270fa5b12c2489abe94f87b4539e7bc" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 44009 } }
I20260928 21:54:36.436110 10792 raft_consensus.cc:374] T 56299f48ff1640659913dee3cb63787a P 05d1046a70f84d86a215b76dc85ef109 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:54:36.436128 10792 raft_consensus.cc:729] T 56299f48ff1640659913dee3cb63787a P 05d1046a70f84d86a215b76dc85ef109 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05d1046a70f84d86a215b76dc85ef109, State: Initialized, Role: FOLLOWER
I20260928 21:54:36.436167 10792 consensus_queue.cc:260] T 56299f48ff1640659913dee3cb63787a P 05d1046a70f84d86a215b76dc85ef109 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c5ce1c5850d406fbd3f4eeca6fde124" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35485 } } peers { permanent_uuid: "05d1046a70f84d86a215b76dc85ef109" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 39201 } } peers { permanent_uuid: "c270fa5b12c2489abe94f87b4539e7bc" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 44009 } }
I20260928 21:54:36.436265 10792 ts_tablet_manager.cc:1428] T 56299f48ff1640659913dee3cb63787a P 05d1046a70f84d86a215b76dc85ef109: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 21:54:36.491891 10846 raft_consensus.cc:482] T 56299f48ff1640659913dee3cb63787a P 05d1046a70f84d86a215b76dc85ef109 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:54:36.492003 10846 raft_consensus.cc:504] T 56299f48ff1640659913dee3cb63787a P 05d1046a70f84d86a215b76dc85ef109 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c5ce1c5850d406fbd3f4eeca6fde124" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35485 } } peers { permanent_uuid: "05d1046a70f84d86a215b76dc85ef109" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 39201 } } peers { permanent_uuid: "c270fa5b12c2489abe94f87b4539e7bc" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 44009 } }
I20260928 21:54:36.492158 10846 leader_election.cc:288] T 56299f48ff1640659913dee3cb63787a P 05d1046a70f84d86a215b76dc85ef109 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c5ce1c5850d406fbd3f4eeca6fde124 (127.2.182.65:35485), c270fa5b12c2489abe94f87b4539e7bc (127.2.182.67:44009)
I20260928 21:54:36.492408 10438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56299f48ff1640659913dee3cb63787a" candidate_uuid: "05d1046a70f84d86a215b76dc85ef109" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c5ce1c5850d406fbd3f4eeca6fde124" is_pre_election: true
I20260928 21:54:36.492450 10702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56299f48ff1640659913dee3cb63787a" candidate_uuid: "05d1046a70f84d86a215b76dc85ef109" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c270fa5b12c2489abe94f87b4539e7bc" is_pre_election: true
I20260928 21:54:36.492514 10438 raft_consensus.cc:2454] T 56299f48ff1640659913dee3cb63787a P 8c5ce1c5850d406fbd3f4eeca6fde124 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05d1046a70f84d86a215b76dc85ef109 in term 0.
I20260928 21:54:36.492528 10702 raft_consensus.cc:2454] T 56299f48ff1640659913dee3cb63787a P c270fa5b12c2489abe94f87b4539e7bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05d1046a70f84d86a215b76dc85ef109 in term 0.
I20260928 21:54:36.492714 10510 leader_election.cc:302] T 56299f48ff1640659913dee3cb63787a P 05d1046a70f84d86a215b76dc85ef109 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05d1046a70f84d86a215b76dc85ef109, 8c5ce1c5850d406fbd3f4eeca6fde124; no voters:
I20260928 21:54:36.492914 10846 raft_consensus.cc:2781] T 56299f48ff1640659913dee3cb63787a P 05d1046a70f84d86a215b76dc85ef109 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:54:36.492956 10846 raft_consensus.cc:482] T 56299f48ff1640659913dee3cb63787a P 05d1046a70f84d86a215b76dc85ef109 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:54:36.492975 10846 raft_consensus.cc:3037] T 56299f48ff1640659913dee3cb63787a P 05d1046a70f84d86a215b76dc85ef109 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:54:36.493643 10846 raft_consensus.cc:504] T 56299f48ff1640659913dee3cb63787a P 05d1046a70f84d86a215b76dc85ef109 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c5ce1c5850d406fbd3f4eeca6fde124" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35485 } } peers { permanent_uuid: "05d1046a70f84d86a215b76dc85ef109" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 39201 } } peers { permanent_uuid: "c270fa5b12c2489abe94f87b4539e7bc" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 44009 } }
I20260928 21:54:36.493768 10846 leader_election.cc:288] T 56299f48ff1640659913dee3cb63787a P 05d1046a70f84d86a215b76dc85ef109 [CANDIDATE]: Term 1 election: Requested vote from peers 8c5ce1c5850d406fbd3f4eeca6fde124 (127.2.182.65:35485), c270fa5b12c2489abe94f87b4539e7bc (127.2.182.67:44009)
I20260928 21:54:36.493961 10702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56299f48ff1640659913dee3cb63787a" candidate_uuid: "05d1046a70f84d86a215b76dc85ef109" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c270fa5b12c2489abe94f87b4539e7bc"
I20260928 21:54:36.493965 10438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56299f48ff1640659913dee3cb63787a" candidate_uuid: "05d1046a70f84d86a215b76dc85ef109" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c5ce1c5850d406fbd3f4eeca6fde124"
I20260928 21:54:36.494035 10702 raft_consensus.cc:3037] T 56299f48ff1640659913dee3cb63787a P c270fa5b12c2489abe94f87b4539e7bc [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:54:36.494040 10438 raft_consensus.cc:3037] T 56299f48ff1640659913dee3cb63787a P 8c5ce1c5850d406fbd3f4eeca6fde124 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:54:36.494597 10702 raft_consensus.cc:2454] T 56299f48ff1640659913dee3cb63787a P c270fa5b12c2489abe94f87b4539e7bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05d1046a70f84d86a215b76dc85ef109 in term 1.
I20260928 21:54:36.494758 10510 leader_election.cc:302] T 56299f48ff1640659913dee3cb63787a P 05d1046a70f84d86a215b76dc85ef109 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05d1046a70f84d86a215b76dc85ef109, c270fa5b12c2489abe94f87b4539e7bc; no voters:
I20260928 21:54:36.494831 10438 raft_consensus.cc:2454] T 56299f48ff1640659913dee3cb63787a P 8c5ce1c5850d406fbd3f4eeca6fde124 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05d1046a70f84d86a215b76dc85ef109 in term 1.
I20260928 21:54:36.494845 10846 raft_consensus.cc:2781] T 56299f48ff1640659913dee3cb63787a P 05d1046a70f84d86a215b76dc85ef109 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:54:36.494889 10846 raft_consensus.cc:686] T 56299f48ff1640659913dee3cb63787a P 05d1046a70f84d86a215b76dc85ef109 [term 1 LEADER]: Becoming Leader. State: Replica: 05d1046a70f84d86a215b76dc85ef109, State: Running, Role: LEADER
I20260928 21:54:36.494938 10846 consensus_queue.cc:237] T 56299f48ff1640659913dee3cb63787a P 05d1046a70f84d86a215b76dc85ef109 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c5ce1c5850d406fbd3f4eeca6fde124" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35485 } } peers { permanent_uuid: "05d1046a70f84d86a215b76dc85ef109" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 39201 } } peers { permanent_uuid: "c270fa5b12c2489abe94f87b4539e7bc" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 44009 } }
I20260928 21:54:36.495553 10281 catalog_manager.cc:5495] T 56299f48ff1640659913dee3cb63787a P 05d1046a70f84d86a215b76dc85ef109 reported cstate change: term changed from 0 to 1, leader changed from <none> to 05d1046a70f84d86a215b76dc85ef109 (127.2.182.66). New cstate: current_term: 1 leader_uuid: "05d1046a70f84d86a215b76dc85ef109" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c5ce1c5850d406fbd3f4eeca6fde124" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05d1046a70f84d86a215b76dc85ef109" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 39201 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c270fa5b12c2489abe94f87b4539e7bc" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 44009 } health_report { overall_health: UNKNOWN } } }
I20260928 21:54:36.508955 10789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:36.506738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34268 (local address 127.2.182.66:39201)
0928 21:54:36.506834 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:36.506836 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:36.506843 (+ 7us) server_negotiation.cc:407] Connection header received
0928 21:54:36.506867 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:36.506867 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:36.506876 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:36.506918 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:54:36.507049 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:36.507615 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:36.508361 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:36.508554 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:36.508608 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:36.508736 (+ 128us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:54:36.508764 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:54:36.508802 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:36.508857 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 21:54:36.508878 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 21:54:36.509824 10702 raft_consensus.cc:1261] T 56299f48ff1640659913dee3cb63787a P c270fa5b12c2489abe94f87b4539e7bc [term 1 FOLLOWER]: Refusing update from remote peer 05d1046a70f84d86a215b76dc85ef109: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:54:36.509837 10438 raft_consensus.cc:1261] T 56299f48ff1640659913dee3cb63787a P 8c5ce1c5850d406fbd3f4eeca6fde124 [term 1 FOLLOWER]: Refusing update from remote peer 05d1046a70f84d86a215b76dc85ef109: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:54:36.510046 10846 consensus_queue.cc:1035] T 56299f48ff1640659913dee3cb63787a P 05d1046a70f84d86a215b76dc85ef109 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c5ce1c5850d406fbd3f4eeca6fde124" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:54:36.510159 10836 consensus_queue.cc:1035] T 56299f48ff1640659913dee3cb63787a P 05d1046a70f84d86a215b76dc85ef109 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c270fa5b12c2489abe94f87b4539e7bc" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 44009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:54:36.513166 10279 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:54:36.513274 10279 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:54:36.514189 10279 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 85cac027afc849588f7eae9726ee1eb1: Sending DeleteTablet for 3 replicas of tablet 56299f48ff1640659913dee3cb63787a
I20260928 21:54:36.514496 10558 tablet_service.cc:1514] Processing DeleteTablet for tablet 56299f48ff1640659913dee3cb63787a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:54:36 UTC) from {username='oryx', principal='oryx/127.2.182.126@KRBTEST.COM'} at 127.0.0.1:34258
I20260928 21:54:36.514567 10418 tablet_service.cc:1514] Processing DeleteTablet for tablet 56299f48ff1640659913dee3cb63787a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:54:36 UTC) from {username='oryx', principal='oryx/127.2.182.126@KRBTEST.COM'} at 127.0.0.1:56518
I20260928 21:54:36.514647 10816 tablet_replica.cc:330] T 56299f48ff1640659913dee3cb63787a P 05d1046a70f84d86a215b76dc85ef109: stopping tablet replica
I20260928 21:54:36.514658 10817 tablet_replica.cc:330] T 56299f48ff1640659913dee3cb63787a P 8c5ce1c5850d406fbd3f4eeca6fde124: stopping tablet replica
I20260928 21:54:36.514703 10817 raft_consensus.cc:2229] T 56299f48ff1640659913dee3cb63787a P 8c5ce1c5850d406fbd3f4eeca6fde124 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:54:36.514703 10816 raft_consensus.cc:2229] T 56299f48ff1640659913dee3cb63787a P 05d1046a70f84d86a215b76dc85ef109 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:54:36.514756 10817 raft_consensus.cc:2258] T 56299f48ff1640659913dee3cb63787a P 8c5ce1c5850d406fbd3f4eeca6fde124 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:54:36.514806 10816 raft_consensus.cc:2258] T 56299f48ff1640659913dee3cb63787a P 05d1046a70f84d86a215b76dc85ef109 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:54:36.514882 10698 tablet_service.cc:1514] Processing DeleteTablet for tablet 56299f48ff1640659913dee3cb63787a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:54:36 UTC) from {username='oryx', principal='oryx/127.2.182.126@KRBTEST.COM'} at 127.0.0.1:40838
I20260928 21:54:36.515026 10815 tablet_replica.cc:330] T 56299f48ff1640659913dee3cb63787a P c270fa5b12c2489abe94f87b4539e7bc: stopping tablet replica
I20260928 21:54:36.515079 10815 raft_consensus.cc:2229] T 56299f48ff1640659913dee3cb63787a P c270fa5b12c2489abe94f87b4539e7bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:54:36.515120 10815 raft_consensus.cc:2258] T 56299f48ff1640659913dee3cb63787a P c270fa5b12c2489abe94f87b4539e7bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:54:36.515044 10279 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43132:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:54:36.515174 10279 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:54:36.515448 10816 ts_tablet_manager.cc:1905] T 56299f48ff1640659913dee3cb63787a P 05d1046a70f84d86a215b76dc85ef109: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:54:36.515455 10815 ts_tablet_manager.cc:1905] T 56299f48ff1640659913dee3cb63787a P c270fa5b12c2489abe94f87b4539e7bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:54:36.516772 10817 ts_tablet_manager.cc:1905] T 56299f48ff1640659913dee3cb63787a P 8c5ce1c5850d406fbd3f4eeca6fde124: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:54:36.516996 10816 ts_tablet_manager.cc:1918] T 56299f48ff1640659913dee3cb63787a P 05d1046a70f84d86a215b76dc85ef109: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:54:36.517045 10816 log.cc:1196] T 56299f48ff1640659913dee3cb63787a P 05d1046a70f84d86a215b76dc85ef109: Deleting WAL directory at /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/minicluster-data/ts-1/wal/wals/56299f48ff1640659913dee3cb63787a
I20260928 21:54:36.517227 10816 ts_tablet_manager.cc:1939] T 56299f48ff1640659913dee3cb63787a P 05d1046a70f84d86a215b76dc85ef109: Deleting consensus metadata
I20260928 21:54:36.517342 10815 ts_tablet_manager.cc:1918] T 56299f48ff1640659913dee3cb63787a P c270fa5b12c2489abe94f87b4539e7bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:54:36.517393 10815 log.cc:1196] T 56299f48ff1640659913dee3cb63787a P c270fa5b12c2489abe94f87b4539e7bc: Deleting WAL directory at /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/minicluster-data/ts-2/wal/wals/56299f48ff1640659913dee3cb63787a
I20260928 21:54:36.517604 10815 ts_tablet_manager.cc:1939] T 56299f48ff1640659913dee3cb63787a P c270fa5b12c2489abe94f87b4539e7bc: Deleting consensus metadata
I20260928 21:54:36.517951 10266 catalog_manager.cc:4841] TS c270fa5b12c2489abe94f87b4539e7bc (127.2.182.67:44009): tablet 56299f48ff1640659913dee3cb63787a (table test-table [id=af5fd92e7eff461499a517588f7f0020]) successfully deleted
I20260928 21:54:36.518213 10817 ts_tablet_manager.cc:1918] T 56299f48ff1640659913dee3cb63787a P 8c5ce1c5850d406fbd3f4eeca6fde124: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:54:36.518261 10817 log.cc:1196] T 56299f48ff1640659913dee3cb63787a P 8c5ce1c5850d406fbd3f4eeca6fde124: Deleting WAL directory at /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/minicluster-data/ts-0/wal/wals/56299f48ff1640659913dee3cb63787a
I20260928 21:54:36.518429 10817 ts_tablet_manager.cc:1939] T 56299f48ff1640659913dee3cb63787a P 8c5ce1c5850d406fbd3f4eeca6fde124: Deleting consensus metadata
I20260928 21:54:36.518780 10267 catalog_manager.cc:4841] TS 05d1046a70f84d86a215b76dc85ef109 (127.2.182.66:39201): tablet 56299f48ff1640659913dee3cb63787a (table test-table [id=af5fd92e7eff461499a517588f7f0020]) successfully deleted
I20260928 21:54:36.519102 10266 catalog_manager.cc:4841] TS 8c5ce1c5850d406fbd3f4eeca6fde124 (127.2.182.65:35485): tablet 56299f48ff1640659913dee3cb63787a (table test-table [id=af5fd92e7eff461499a517588f7f0020]) successfully deleted
I20260928 21:54:36.519292 10558 tablet_service.cc:1467] Processing CreateTablet for tablet 1fc9e108883b45f49e30db4621392530 (DEFAULT_TABLE table=test-table [id=c7f069bb148d4d17b668468d003cb875]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:54:36.519318 10418 tablet_service.cc:1467] Processing CreateTablet for tablet 1fc9e108883b45f49e30db4621392530 (DEFAULT_TABLE table=test-table [id=c7f069bb148d4d17b668468d003cb875]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:54:36.519397 10698 tablet_service.cc:1467] Processing CreateTablet for tablet 1fc9e108883b45f49e30db4621392530 (DEFAULT_TABLE table=test-table [id=c7f069bb148d4d17b668468d003cb875]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:54:36.519425 10418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fc9e108883b45f49e30db4621392530. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:54:36.519424 10558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fc9e108883b45f49e30db4621392530. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:54:36.519493 10698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fc9e108883b45f49e30db4621392530. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:54:36.520627 10793 tablet_bootstrap.cc:492] T 1fc9e108883b45f49e30db4621392530 P 8c5ce1c5850d406fbd3f4eeca6fde124: Bootstrap starting.
I20260928 21:54:36.520628 10792 tablet_bootstrap.cc:492] T 1fc9e108883b45f49e30db4621392530 P 05d1046a70f84d86a215b76dc85ef109: Bootstrap starting.
I20260928 21:54:36.520902 10792 tablet_bootstrap.cc:654] T 1fc9e108883b45f49e30db4621392530 P 05d1046a70f84d86a215b76dc85ef109: Neither blocks nor log segments found. Creating new log.
I20260928 21:54:36.520922 10793 tablet_bootstrap.cc:654] T 1fc9e108883b45f49e30db4621392530 P 8c5ce1c5850d406fbd3f4eeca6fde124: Neither blocks nor log segments found. Creating new log.
I20260928 21:54:36.521240 10796 tablet_bootstrap.cc:492] T 1fc9e108883b45f49e30db4621392530 P c270fa5b12c2489abe94f87b4539e7bc: Bootstrap starting.
I20260928 21:54:36.521382 10792 tablet_bootstrap.cc:492] T 1fc9e108883b45f49e30db4621392530 P 05d1046a70f84d86a215b76dc85ef109: No bootstrap required, opened a new log
I20260928 21:54:36.521428 10792 ts_tablet_manager.cc:1397] T 1fc9e108883b45f49e30db4621392530 P 05d1046a70f84d86a215b76dc85ef109: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260928 21:54:36.521503 10796 tablet_bootstrap.cc:654] T 1fc9e108883b45f49e30db4621392530 P c270fa5b12c2489abe94f87b4539e7bc: Neither blocks nor log segments found. Creating new log.
I20260928 21:54:36.521534 10792 raft_consensus.cc:348] T 1fc9e108883b45f49e30db4621392530 P 05d1046a70f84d86a215b76dc85ef109 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c5ce1c5850d406fbd3f4eeca6fde124" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35485 } } peers { permanent_uuid: "05d1046a70f84d86a215b76dc85ef109" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 39201 } } peers { permanent_uuid: "c270fa5b12c2489abe94f87b4539e7bc" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 44009 } }
I20260928 21:54:36.521606 10792 raft_consensus.cc:374] T 1fc9e108883b45f49e30db4621392530 P 05d1046a70f84d86a215b76dc85ef109 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:54:36.521627 10792 raft_consensus.cc:729] T 1fc9e108883b45f49e30db4621392530 P 05d1046a70f84d86a215b76dc85ef109 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05d1046a70f84d86a215b76dc85ef109, State: Initialized, Role: FOLLOWER
I20260928 21:54:36.521667 10792 consensus_queue.cc:260] T 1fc9e108883b45f49e30db4621392530 P 05d1046a70f84d86a215b76dc85ef109 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c5ce1c5850d406fbd3f4eeca6fde124" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35485 } } peers { permanent_uuid: "05d1046a70f84d86a215b76dc85ef109" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 39201 } } peers { permanent_uuid: "c270fa5b12c2489abe94f87b4539e7bc" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 44009 } }
I20260928 21:54:36.521756 10792 ts_tablet_manager.cc:1428] T 1fc9e108883b45f49e30db4621392530 P 05d1046a70f84d86a215b76dc85ef109: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:36.522147 10793 tablet_bootstrap.cc:492] T 1fc9e108883b45f49e30db4621392530 P 8c5ce1c5850d406fbd3f4eeca6fde124: No bootstrap required, opened a new log
I20260928 21:54:36.522199 10793 ts_tablet_manager.cc:1397] T 1fc9e108883b45f49e30db4621392530 P 8c5ce1c5850d406fbd3f4eeca6fde124: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:54:36.522306 10793 raft_consensus.cc:348] T 1fc9e108883b45f49e30db4621392530 P 8c5ce1c5850d406fbd3f4eeca6fde124 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c5ce1c5850d406fbd3f4eeca6fde124" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35485 } } peers { permanent_uuid: "05d1046a70f84d86a215b76dc85ef109" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 39201 } } peers { permanent_uuid: "c270fa5b12c2489abe94f87b4539e7bc" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 44009 } }
I20260928 21:54:36.522375 10793 raft_consensus.cc:374] T 1fc9e108883b45f49e30db4621392530 P 8c5ce1c5850d406fbd3f4eeca6fde124 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:54:36.522394 10793 raft_consensus.cc:729] T 1fc9e108883b45f49e30db4621392530 P 8c5ce1c5850d406fbd3f4eeca6fde124 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c5ce1c5850d406fbd3f4eeca6fde124, State: Initialized, Role: FOLLOWER
I20260928 21:54:36.522439 10793 consensus_queue.cc:260] T 1fc9e108883b45f49e30db4621392530 P 8c5ce1c5850d406fbd3f4eeca6fde124 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c5ce1c5850d406fbd3f4eeca6fde124" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35485 } } peers { permanent_uuid: "05d1046a70f84d86a215b76dc85ef109" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 39201 } } peers { permanent_uuid: "c270fa5b12c2489abe94f87b4539e7bc" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 44009 } }
I20260928 21:54:36.522526 10793 ts_tablet_manager.cc:1428] T 1fc9e108883b45f49e30db4621392530 P 8c5ce1c5850d406fbd3f4eeca6fde124: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:36.522723 10796 tablet_bootstrap.cc:492] T 1fc9e108883b45f49e30db4621392530 P c270fa5b12c2489abe94f87b4539e7bc: No bootstrap required, opened a new log
I20260928 21:54:36.522775 10796 ts_tablet_manager.cc:1397] T 1fc9e108883b45f49e30db4621392530 P c270fa5b12c2489abe94f87b4539e7bc: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s
I20260928 21:54:36.522884 10796 raft_consensus.cc:348] T 1fc9e108883b45f49e30db4621392530 P c270fa5b12c2489abe94f87b4539e7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c5ce1c5850d406fbd3f4eeca6fde124" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35485 } } peers { permanent_uuid: "05d1046a70f84d86a215b76dc85ef109" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 39201 } } peers { permanent_uuid: "c270fa5b12c2489abe94f87b4539e7bc" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 44009 } }
I20260928 21:54:36.522943 10796 raft_consensus.cc:374] T 1fc9e108883b45f49e30db4621392530 P c270fa5b12c2489abe94f87b4539e7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:54:36.522959 10796 raft_consensus.cc:729] T 1fc9e108883b45f49e30db4621392530 P c270fa5b12c2489abe94f87b4539e7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c270fa5b12c2489abe94f87b4539e7bc, State: Initialized, Role: FOLLOWER
I20260928 21:54:36.523000 10796 consensus_queue.cc:260] T 1fc9e108883b45f49e30db4621392530 P c270fa5b12c2489abe94f87b4539e7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c5ce1c5850d406fbd3f4eeca6fde124" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35485 } } peers { permanent_uuid: "05d1046a70f84d86a215b76dc85ef109" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 39201 } } peers { permanent_uuid: "c270fa5b12c2489abe94f87b4539e7bc" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 44009 } }
I20260928 21:54:36.523075 10796 ts_tablet_manager.cc:1428] T 1fc9e108883b45f49e30db4621392530 P c270fa5b12c2489abe94f87b4539e7bc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:36.795228 10823 raft_consensus.cc:482] T 1fc9e108883b45f49e30db4621392530 P 8c5ce1c5850d406fbd3f4eeca6fde124 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:54:36.795370 10823 raft_consensus.cc:504] T 1fc9e108883b45f49e30db4621392530 P 8c5ce1c5850d406fbd3f4eeca6fde124 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c5ce1c5850d406fbd3f4eeca6fde124" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35485 } } peers { permanent_uuid: "05d1046a70f84d86a215b76dc85ef109" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 39201 } } peers { permanent_uuid: "c270fa5b12c2489abe94f87b4539e7bc" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 44009 } }
I20260928 21:54:36.795522 10823 leader_election.cc:288] T 1fc9e108883b45f49e30db4621392530 P 8c5ce1c5850d406fbd3f4eeca6fde124 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05d1046a70f84d86a215b76dc85ef109 (127.2.182.66:39201), c270fa5b12c2489abe94f87b4539e7bc (127.2.182.67:44009)
I20260928 21:54:36.795739 10702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fc9e108883b45f49e30db4621392530" candidate_uuid: "8c5ce1c5850d406fbd3f4eeca6fde124" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c270fa5b12c2489abe94f87b4539e7bc" is_pre_election: true
I20260928 21:54:36.795827 10702 raft_consensus.cc:2454] T 1fc9e108883b45f49e30db4621392530 P c270fa5b12c2489abe94f87b4539e7bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c5ce1c5850d406fbd3f4eeca6fde124 in term 0.
I20260928 21:54:36.795818 10576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fc9e108883b45f49e30db4621392530" candidate_uuid: "8c5ce1c5850d406fbd3f4eeca6fde124" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05d1046a70f84d86a215b76dc85ef109" is_pre_election: true
I20260928 21:54:36.795917 10576 raft_consensus.cc:2454] T 1fc9e108883b45f49e30db4621392530 P 05d1046a70f84d86a215b76dc85ef109 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c5ce1c5850d406fbd3f4eeca6fde124 in term 0.
I20260928 21:54:36.796023 10370 leader_election.cc:302] T 1fc9e108883b45f49e30db4621392530 P 8c5ce1c5850d406fbd3f4eeca6fde124 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c5ce1c5850d406fbd3f4eeca6fde124, c270fa5b12c2489abe94f87b4539e7bc; no voters:
I20260928 21:54:36.796123 10823 raft_consensus.cc:2781] T 1fc9e108883b45f49e30db4621392530 P 8c5ce1c5850d406fbd3f4eeca6fde124 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:54:36.796166 10823 raft_consensus.cc:482] T 1fc9e108883b45f49e30db4621392530 P 8c5ce1c5850d406fbd3f4eeca6fde124 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:54:36.796180 10823 raft_consensus.cc:3037] T 1fc9e108883b45f49e30db4621392530 P 8c5ce1c5850d406fbd3f4eeca6fde124 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:54:36.796828 10823 raft_consensus.cc:504] T 1fc9e108883b45f49e30db4621392530 P 8c5ce1c5850d406fbd3f4eeca6fde124 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c5ce1c5850d406fbd3f4eeca6fde124" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35485 } } peers { permanent_uuid: "05d1046a70f84d86a215b76dc85ef109" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 39201 } } peers { permanent_uuid: "c270fa5b12c2489abe94f87b4539e7bc" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 44009 } }
I20260928 21:54:36.796944 10823 leader_election.cc:288] T 1fc9e108883b45f49e30db4621392530 P 8c5ce1c5850d406fbd3f4eeca6fde124 [CANDIDATE]: Term 1 election: Requested vote from peers 05d1046a70f84d86a215b76dc85ef109 (127.2.182.66:39201), c270fa5b12c2489abe94f87b4539e7bc (127.2.182.67:44009)
I20260928 21:54:36.797144 10576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fc9e108883b45f49e30db4621392530" candidate_uuid: "8c5ce1c5850d406fbd3f4eeca6fde124" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05d1046a70f84d86a215b76dc85ef109"
I20260928 21:54:36.797138 10702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fc9e108883b45f49e30db4621392530" candidate_uuid: "8c5ce1c5850d406fbd3f4eeca6fde124" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c270fa5b12c2489abe94f87b4539e7bc"
I20260928 21:54:36.797219 10702 raft_consensus.cc:3037] T 1fc9e108883b45f49e30db4621392530 P c270fa5b12c2489abe94f87b4539e7bc [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:54:36.797219 10576 raft_consensus.cc:3037] T 1fc9e108883b45f49e30db4621392530 P 05d1046a70f84d86a215b76dc85ef109 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:54:36.797957 10576 raft_consensus.cc:2454] T 1fc9e108883b45f49e30db4621392530 P 05d1046a70f84d86a215b76dc85ef109 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c5ce1c5850d406fbd3f4eeca6fde124 in term 1.
I20260928 21:54:36.797957 10702 raft_consensus.cc:2454] T 1fc9e108883b45f49e30db4621392530 P c270fa5b12c2489abe94f87b4539e7bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c5ce1c5850d406fbd3f4eeca6fde124 in term 1.
I20260928 21:54:36.798161 10371 leader_election.cc:302] T 1fc9e108883b45f49e30db4621392530 P 8c5ce1c5850d406fbd3f4eeca6fde124 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05d1046a70f84d86a215b76dc85ef109, 8c5ce1c5850d406fbd3f4eeca6fde124; no voters:
I20260928 21:54:36.798328 10823 raft_consensus.cc:2781] T 1fc9e108883b45f49e30db4621392530 P 8c5ce1c5850d406fbd3f4eeca6fde124 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:54:36.798388 10823 raft_consensus.cc:686] T 1fc9e108883b45f49e30db4621392530 P 8c5ce1c5850d406fbd3f4eeca6fde124 [term 1 LEADER]: Becoming Leader. State: Replica: 8c5ce1c5850d406fbd3f4eeca6fde124, State: Running, Role: LEADER
I20260928 21:54:36.798435 10823 consensus_queue.cc:237] T 1fc9e108883b45f49e30db4621392530 P 8c5ce1c5850d406fbd3f4eeca6fde124 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c5ce1c5850d406fbd3f4eeca6fde124" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35485 } } peers { permanent_uuid: "05d1046a70f84d86a215b76dc85ef109" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 39201 } } peers { permanent_uuid: "c270fa5b12c2489abe94f87b4539e7bc" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 44009 } }
I20260928 21:54:36.799016 10279 catalog_manager.cc:5495] T 1fc9e108883b45f49e30db4621392530 P 8c5ce1c5850d406fbd3f4eeca6fde124 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c5ce1c5850d406fbd3f4eeca6fde124 (127.2.182.65). New cstate: current_term: 1 leader_uuid: "8c5ce1c5850d406fbd3f4eeca6fde124" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c5ce1c5850d406fbd3f4eeca6fde124" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05d1046a70f84d86a215b76dc85ef109" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 39201 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c270fa5b12c2489abe94f87b4539e7bc" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 44009 } health_report { overall_health: UNKNOWN } } }
I20260928 21:54:36.806428 10791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:36.804354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56544 (local address 127.2.182.65:35485)
0928 21:54:36.804393 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:36.804395 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:36.804402 (+ 7us) server_negotiation.cc:407] Connection header received
0928 21:54:36.804470 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:36.804470 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:36.804478 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:36.804517 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:54:36.804599 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:36.805057 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:36.805767 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:36.805963 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:36.806025 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:36.806209 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:54:36.806241 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:54:36.806287 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:36.806335 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 21:54:36.806355 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 21:54:36.808507 10702 raft_consensus.cc:1261] T 1fc9e108883b45f49e30db4621392530 P c270fa5b12c2489abe94f87b4539e7bc [term 1 FOLLOWER]: Refusing update from remote peer 8c5ce1c5850d406fbd3f4eeca6fde124: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:54:36.808564 10576 raft_consensus.cc:1261] T 1fc9e108883b45f49e30db4621392530 P 05d1046a70f84d86a215b76dc85ef109 [term 1 FOLLOWER]: Refusing update from remote peer 8c5ce1c5850d406fbd3f4eeca6fde124: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:54:36.808754 10823 consensus_queue.cc:1035] T 1fc9e108883b45f49e30db4621392530 P 8c5ce1c5850d406fbd3f4eeca6fde124 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05d1046a70f84d86a215b76dc85ef109" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 39201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:54:36.808858 10849 consensus_queue.cc:1035] T 1fc9e108883b45f49e30db4621392530 P 8c5ce1c5850d406fbd3f4eeca6fde124 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c270fa5b12c2489abe94f87b4539e7bc" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 44009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:54:36.820086 10281 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:54:36.820194 10281 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:54:36.820820 10281 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 85cac027afc849588f7eae9726ee1eb1: Sending DeleteTablet for 3 replicas of tablet 1fc9e108883b45f49e30db4621392530
I20260928 21:54:36.821077 10418 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fc9e108883b45f49e30db4621392530 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:54:36 UTC) from {username='oryx', principal='oryx/127.2.182.126@KRBTEST.COM'} at 127.0.0.1:56518
I20260928 21:54:36.821091 10558 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fc9e108883b45f49e30db4621392530 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:54:36 UTC) from {username='oryx', principal='oryx/127.2.182.126@KRBTEST.COM'} at 127.0.0.1:34258
I20260928 21:54:36.821195 10817 tablet_replica.cc:330] T 1fc9e108883b45f49e30db4621392530 P 8c5ce1c5850d406fbd3f4eeca6fde124: stopping tablet replica
I20260928 21:54:36.821189 10698 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fc9e108883b45f49e30db4621392530 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:54:36 UTC) from {username='oryx', principal='oryx/127.2.182.126@KRBTEST.COM'} at 127.0.0.1:40838
I20260928 21:54:36.821257 10817 raft_consensus.cc:2229] T 1fc9e108883b45f49e30db4621392530 P 8c5ce1c5850d406fbd3f4eeca6fde124 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:54:36.821290 10815 tablet_replica.cc:330] T 1fc9e108883b45f49e30db4621392530 P c270fa5b12c2489abe94f87b4539e7bc: stopping tablet replica
I20260928 21:54:36.821327 10817 raft_consensus.cc:2258] T 1fc9e108883b45f49e30db4621392530 P 8c5ce1c5850d406fbd3f4eeca6fde124 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:54:36.821354 10815 raft_consensus.cc:2229] T 1fc9e108883b45f49e30db4621392530 P c270fa5b12c2489abe94f87b4539e7bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:54:36.821408 10815 raft_consensus.cc:2258] T 1fc9e108883b45f49e30db4621392530 P c270fa5b12c2489abe94f87b4539e7bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:54:36.821401 10816 tablet_replica.cc:330] T 1fc9e108883b45f49e30db4621392530 P 05d1046a70f84d86a215b76dc85ef109: stopping tablet replica
I20260928 21:54:36.821465 10816 raft_consensus.cc:2229] T 1fc9e108883b45f49e30db4621392530 P 05d1046a70f84d86a215b76dc85ef109 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:54:36.821511 10816 raft_consensus.cc:2258] T 1fc9e108883b45f49e30db4621392530 P 05d1046a70f84d86a215b76dc85ef109 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:54:36.821638 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 10353
I20260928 21:54:36.821789 10817 ts_tablet_manager.cc:1905] T 1fc9e108883b45f49e30db4621392530 P 8c5ce1c5850d406fbd3f4eeca6fde124: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:54:36.821796 10816 ts_tablet_manager.cc:1905] T 1fc9e108883b45f49e30db4621392530 P 05d1046a70f84d86a215b76dc85ef109: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:54:36.822315 10815 ts_tablet_manager.cc:1905] T 1fc9e108883b45f49e30db4621392530 P c270fa5b12c2489abe94f87b4539e7bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:54:36.823434 10816 ts_tablet_manager.cc:1918] T 1fc9e108883b45f49e30db4621392530 P 05d1046a70f84d86a215b76dc85ef109: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:54:36.823490 10816 log.cc:1196] T 1fc9e108883b45f49e30db4621392530 P 05d1046a70f84d86a215b76dc85ef109: Deleting WAL directory at /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/minicluster-data/ts-1/wal/wals/1fc9e108883b45f49e30db4621392530
I20260928 21:54:36.823696 10815 ts_tablet_manager.cc:1918] T 1fc9e108883b45f49e30db4621392530 P c270fa5b12c2489abe94f87b4539e7bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:54:36.823714 10816 ts_tablet_manager.cc:1939] T 1fc9e108883b45f49e30db4621392530 P 05d1046a70f84d86a215b76dc85ef109: Deleting consensus metadata
I20260928 21:54:36.823741 10815 log.cc:1196] T 1fc9e108883b45f49e30db4621392530 P c270fa5b12c2489abe94f87b4539e7bc: Deleting WAL directory at /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632440055392-2777-0/minicluster-data/ts-2/wal/wals/1fc9e108883b45f49e30db4621392530
I20260928 21:54:36.823944 10815 ts_tablet_manager.cc:1939] T 1fc9e108883b45f49e30db4621392530 P c270fa5b12c2489abe94f87b4539e7bc: Deleting consensus metadata
I20260928 21:54:36.824254 10267 catalog_manager.cc:4841] TS 05d1046a70f84d86a215b76dc85ef109 (127.2.182.66:39201): tablet 1fc9e108883b45f49e30db4621392530 (table test-table [id=c7f069bb148d4d17b668468d003cb875]) successfully deleted
I20260928 21:54:36.824415 10266 catalog_manager.cc:4841] TS c270fa5b12c2489abe94f87b4539e7bc (127.2.182.67:44009): tablet 1fc9e108883b45f49e30db4621392530 (table test-table [id=c7f069bb148d4d17b668468d003cb875]) successfully deleted
W20260928 21:54:36.827992 10266 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.182.65:35485 (error 108)
I20260928 21:54:36.828122 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 10493
I20260928 21:54:36.829388 10880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:36.829189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.65:35485 (local address 127.0.0.1:56554)
0928 21:54:36.829292 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:36.829341 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.65:35485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":40,"threads_started":1}
W20260928 21:54:36.829643 10266 catalog_manager.cc:4584] TS 8c5ce1c5850d406fbd3f4eeca6fde124 (127.2.182.65:35485): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1fc9e108883b45f49e30db4621392530: Network error: Client connection negotiation failed: client connection to 127.2.182.65:35485: connect: Connection refused (error 111)
I20260928 21:54:36.833256 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 10633
I20260928 21:54:36.838107 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 10249
2026-09-28T21:54:36Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2775 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790632440055392-2777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:54:36 dist-test-slave-k6lz krb5kdc[10883](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:54:36 dist-test-slave-k6lz krb5kdc[10883](info): set up 2 sockets
Sep 28 21:54:36 dist-test-slave-k6lz krb5kdc[10883](info): commencing operation
krb5kdc: starting...
W20260928 21:54:37.795702 2777 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Sep 28 21:54:37 dist-test-slave-k6lz krb5kdc[10883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632477, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:54:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:54:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab.
Entry for principal kudu/127.2.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab.
I20260928 21:54:37.928253 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790632440055392-2777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790632440055392-2777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790632440055392-2777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:37795
--webserver_interface=127.2.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab
--principal=kudu/127.2.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:38989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.126:37795
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:54:38.011413 10896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:54:38.011638 10896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:54:38.011662 10896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:54:38.012888 10896 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:54:38.012935 10896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:54:38.012948 10896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:54:38.012960 10896 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:54:38.012970 10896 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:54:38.014596 10896 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:38989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790632440055392-2777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790632440055392-2777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.126:37795
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:37795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10896
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790632440055392-2777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:54:38.014856 10896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:54:38.015105 10896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:54:38.020889 10901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:38.020938 10896 server_base.cc:1034] running on GCE node
W20260928 21:54:38.021258 10902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:38.021332 10904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:38.021675 10896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:38.021994 10896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:38.023200 10896 hybrid_clock.cc:648] HybridClock initialized: now 1790632478023178 us; error 43 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-28T21:54:38Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1188 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:54:38 dist-test-slave-k6lz krb5kdc[10909](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:54:38 dist-test-slave-k6lz krb5kdc[10909](info): set up 2 sockets
Sep 28 21:54:38 dist-test-slave-k6lz krb5kdc[10909](info): commencing operation
krb5kdc: starting...
W20260928 21:54:38.987747 2777 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Sep 28 21:54:39 dist-test-slave-k6lz krb5kdc[10909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:54:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:54:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab.
Entry for principal kudu/127.2.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab.
I20260928 21:54:39.106703 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:45403
--webserver_interface=127.2.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab
--principal=kudu/127.2.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:34627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.126:45403
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:54:39.185814 10922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:54:39.186038 10922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:54:39.186069 10922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:54:39.187224 10922 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:54:39.187266 10922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:54:39.187284 10922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:54:39.187300 10922 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:54:39.187316 10922 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:54:39.188835 10922 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:34627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.126:45403
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab
--principal=kudu/127.2.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:45403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10922
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:54:39.189090 10922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:54:39.189293 10922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:54:39.193923 10930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:39.193997 10927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:39.194034 10928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:39.194192 10922 server_base.cc:1034] running on GCE node
I20260928 21:54:39.194416 10922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:39.194667 10922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:39.195878 10922 hybrid_clock.cc:648] HybridClock initialized: now 1790632479195791 us; error 30 us; skew 500 ppm
Sep 28 21:54:39 dist-test-slave-k6lz krb5kdc[10909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632479, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:54:39.202435 10922 init.cc:377] Logged in from keytab as kudu/127.2.182.126@KRBTEST.COM (short username kudu)
I20260928 21:54:39.202975 10922 webserver.cc:460] Webserver started at http://127.2.182.126:42003/ using document root <none> and password file <none>
I20260928 21:54:39.203181 10922 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:39.203228 10922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:39.203383 10922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:54:39.204465 10922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/master-0/data/instance:
uuid: "b394a9607b4c4930a4bd9a024700d54f"
format_stamp: "Formatted at 2026-09-28 21:54:39 on dist-test-slave-k6lz"
server_key: "ec2c64f12effa6cce716134ea5eeb7d0"
server_key_iv: "a4db1a76de42161dbb5bec0c7b1cf79a"
server_key_version: "encryptionkey@0"
I20260928 21:54:39.204813 10922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/master-0/wal/instance:
uuid: "b394a9607b4c4930a4bd9a024700d54f"
format_stamp: "Formatted at 2026-09-28 21:54:39 on dist-test-slave-k6lz"
server_key: "ec2c64f12effa6cce716134ea5eeb7d0"
server_key_iv: "a4db1a76de42161dbb5bec0c7b1cf79a"
server_key_version: "encryptionkey@0"
I20260928 21:54:39.206055 10922 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:54:39.206785 10937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:39.207051 10922 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:54:39.207140 10922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/master-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/master-0/wal
uuid: "b394a9607b4c4930a4bd9a024700d54f"
format_stamp: "Formatted at 2026-09-28 21:54:39 on dist-test-slave-k6lz"
server_key: "ec2c64f12effa6cce716134ea5eeb7d0"
server_key_iv: "a4db1a76de42161dbb5bec0c7b1cf79a"
server_key_version: "encryptionkey@0"
I20260928 21:54:39.207201 10922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:39.219105 10922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:54:39.221276 10922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:54:39.221421 10922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:54:39.224956 10922 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.126:45403
I20260928 21:54:39.225013 10989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.126:45403 every 8 connection(s)
I20260928 21:54:39.225340 10922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
I20260928 21:54:39.226155 10990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:54:39.228436 10990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b394a9607b4c4930a4bd9a024700d54f: Bootstrap starting.
I20260928 21:54:39.229004 10990 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b394a9607b4c4930a4bd9a024700d54f: Neither blocks nor log segments found. Creating new log.
I20260928 21:54:39.229269 10990 log.cc:824] T 00000000000000000000000000000000 P b394a9607b4c4930a4bd9a024700d54f: Log is configured to *not* fsync() on all Append() calls
I20260928 21:54:39.229954 10990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b394a9607b4c4930a4bd9a024700d54f: No bootstrap required, opened a new log
I20260928 21:54:39.231593 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 10922
I20260928 21:54:39.231694 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/master-0/wal/instance
I20260928 21:54:39.231662 10990 raft_consensus.cc:348] T 00000000000000000000000000000000 P b394a9607b4c4930a4bd9a024700d54f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b394a9607b4c4930a4bd9a024700d54f" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 45403 } }
I20260928 21:54:39.231845 10990 raft_consensus.cc:374] T 00000000000000000000000000000000 P b394a9607b4c4930a4bd9a024700d54f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:54:39.231894 2777 external_mini_cluster.cc:1428] Setting key c6064edb04d58ce6cd3c39648fc49dfa
I20260928 21:54:39.231894 10990 raft_consensus.cc:729] T 00000000000000000000000000000000 P b394a9607b4c4930a4bd9a024700d54f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b394a9607b4c4930a4bd9a024700d54f, State: Initialized, Role: FOLLOWER
I20260928 21:54:39.232038 10990 consensus_queue.cc:260] T 00000000000000000000000000000000 P b394a9607b4c4930a4bd9a024700d54f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b394a9607b4c4930a4bd9a024700d54f" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 45403 } }
I20260928 21:54:39.232121 10990 raft_consensus.cc:388] T 00000000000000000000000000000000 P b394a9607b4c4930a4bd9a024700d54f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:54:39.232157 10990 raft_consensus.cc:482] T 00000000000000000000000000000000 P b394a9607b4c4930a4bd9a024700d54f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:54:39.232198 10990 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b394a9607b4c4930a4bd9a024700d54f [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:54:39.233253 10990 raft_consensus.cc:504] T 00000000000000000000000000000000 P b394a9607b4c4930a4bd9a024700d54f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b394a9607b4c4930a4bd9a024700d54f" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 45403 } }
I20260928 21:54:39.233388 10990 leader_election.cc:302] T 00000000000000000000000000000000 P b394a9607b4c4930a4bd9a024700d54f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b394a9607b4c4930a4bd9a024700d54f; no voters:
I20260928 21:54:39.233556 10990 leader_election.cc:288] T 00000000000000000000000000000000 P b394a9607b4c4930a4bd9a024700d54f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:54:39.233590 10995 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b394a9607b4c4930a4bd9a024700d54f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:54:39.233731 10995 raft_consensus.cc:686] T 00000000000000000000000000000000 P b394a9607b4c4930a4bd9a024700d54f [term 1 LEADER]: Becoming Leader. State: Replica: b394a9607b4c4930a4bd9a024700d54f, State: Running, Role: LEADER
I20260928 21:54:39.233834 10990 sys_catalog.cc:564] T 00000000000000000000000000000000 P b394a9607b4c4930a4bd9a024700d54f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:54:39.233826 10995 consensus_queue.cc:237] T 00000000000000000000000000000000 P b394a9607b4c4930a4bd9a024700d54f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b394a9607b4c4930a4bd9a024700d54f" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 45403 } }
I20260928 21:54:39.234289 10997 sys_catalog.cc:455] T 00000000000000000000000000000000 P b394a9607b4c4930a4bd9a024700d54f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b394a9607b4c4930a4bd9a024700d54f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b394a9607b4c4930a4bd9a024700d54f" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 45403 } } }
I20260928 21:54:39.234375 10997 sys_catalog.cc:458] T 00000000000000000000000000000000 P b394a9607b4c4930a4bd9a024700d54f [sys.catalog]: This master's current role is: LEADER
I20260928 21:54:39.234557 10996 sys_catalog.cc:455] T 00000000000000000000000000000000 P b394a9607b4c4930a4bd9a024700d54f [sys.catalog]: SysCatalogTable state changed. Reason: New leader b394a9607b4c4930a4bd9a024700d54f. Latest consensus state: current_term: 1 leader_uuid: "b394a9607b4c4930a4bd9a024700d54f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b394a9607b4c4930a4bd9a024700d54f" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 45403 } } }
I20260928 21:54:39.234622 10996 sys_catalog.cc:458] T 00000000000000000000000000000000 P b394a9607b4c4930a4bd9a024700d54f [sys.catalog]: This master's current role is: LEADER
I20260928 21:54:39.234732 11004 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:54:39.235405 11004 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:54:39.237229 11004 catalog_manager.cc:1347] Generated new cluster ID: 2c5b3234f16443fe924a95ee6ba6ce35
I20260928 21:54:39.237284 11004 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 21:54:39 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:54:39.241875 10994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:39.232428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42956 (local address 127.2.182.126:45403)
0928 21:54:39.232607 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:39.232616 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:39.232642 (+ 26us) server_negotiation.cc:407] Connection header received
0928 21:54:39.232746 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:39.232751 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:39.232794 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:39.232971 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:39.233738 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:39.235175 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:39.236103 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:39.236702 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:39.236813 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:39.238958 (+ 2145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:39.238962 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:39.238963 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:39.238975 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:39.240492 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:39.240803 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:39.240806 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:39.240806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:39.240899 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:39.241075 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:39.241076 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:39.241076 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:39.241310 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:39.241476 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:39.241577 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 21:54:39.241715 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":84,"threads_started":1}
I20260928 21:54:39.267017 11004 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:54:39.267833 11004 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:54:39.281344 11004 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b394a9607b4c4930a4bd9a024700d54f: Generated new TSK 0
I20260928 21:54:39.281563 11004 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.182.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab.
Entry for principal kudu/127.2.182.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab.
I20260928 21:54:39.310212 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.65:0
--local_ip_for_outbound_sockets=127.2.182.65
--webserver_interface=127.2.182.65
--webserver_port=0
--tserver_master_addrs=127.2.182.126:45403
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab
--principal=kudu/127.2.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:34627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:54:39.392261 11016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:54:39.392573 11016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:54:39.392640 11016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:54:39.394043 11016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:54:39.394099 11016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:54:39.394138 11016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.65
I20260928 21:54:39.395972 11016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:34627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab
--principal=kudu/127.2.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.182.65
--webserver_port=0
--tserver_master_addrs=127.2.182.126:45403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11016
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.65
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:54:39.396232 11016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:54:39.396467 11016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:54:39.401671 11024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:39.401690 11022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:39.402019 11021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:39.402040 11016 server_base.cc:1034] running on GCE node
I20260928 21:54:39.402292 11016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:39.402590 11016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:39.403756 11016 hybrid_clock.cc:648] HybridClock initialized: now 1790632479403741 us; error 28 us; skew 500 ppm
Sep 28 21:54:39 dist-test-slave-k6lz krb5kdc[10909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632479, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:54:39.406601 11016 init.cc:377] Logged in from keytab as kudu/127.2.182.65@KRBTEST.COM (short username kudu)
I20260928 21:54:39.407121 11016 webserver.cc:460] Webserver started at http://127.2.182.65:40535/ using document root <none> and password file <none>
I20260928 21:54:39.407372 11016 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:39.407424 11016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:39.407557 11016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:54:39.408612 11016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-0/data/instance:
uuid: "60b3505edffb416e9a2baef327b50117"
format_stamp: "Formatted at 2026-09-28 21:54:39 on dist-test-slave-k6lz"
server_key: "57739c7bc1e2eefb73bc26a5764321c0"
server_key_iv: "d08d6fc9a3c045805a40d0674d9bdb60"
server_key_version: "encryptionkey@0"
I20260928 21:54:39.408985 11016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-0/wal/instance:
uuid: "60b3505edffb416e9a2baef327b50117"
format_stamp: "Formatted at 2026-09-28 21:54:39 on dist-test-slave-k6lz"
server_key: "57739c7bc1e2eefb73bc26a5764321c0"
server_key_iv: "d08d6fc9a3c045805a40d0674d9bdb60"
server_key_version: "encryptionkey@0"
I20260928 21:54:39.410281 11016 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 21:54:39.411113 11031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:39.411429 11016 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:54:39.411520 11016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-0/wal
uuid: "60b3505edffb416e9a2baef327b50117"
format_stamp: "Formatted at 2026-09-28 21:54:39 on dist-test-slave-k6lz"
server_key: "57739c7bc1e2eefb73bc26a5764321c0"
server_key_iv: "d08d6fc9a3c045805a40d0674d9bdb60"
server_key_version: "encryptionkey@0"
I20260928 21:54:39.411587 11016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:39.432332 11016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:54:39.434542 11016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:54:39.434705 11016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:54:39.434928 11016 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:54:39.435232 11016 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:54:39.435267 11016 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:39.435297 11016 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:54:39.435313 11016 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:39.441030 11016 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.65:45765
I20260928 21:54:39.441076 11144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.65:45765 every 8 connection(s)
I20260928 21:54:39.441386 11016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
I20260928 21:54:39.445642 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 11016
I20260928 21:54:39.445739 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-0/wal/instance
I20260928 21:54:39.445915 2777 external_mini_cluster.cc:1428] Setting key 7d59b651ebc8c4d159960c8f5c690bea
Sep 28 21:54:39 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632479, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.65@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:54:39.451140 10994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:39.442113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.65:45731 (local address 127.2.182.126:45403)
0928 21:54:39.442164 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:39.442166 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:39.442381 (+ 215us) server_negotiation.cc:407] Connection header received
0928 21:54:39.442799 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:39.442800 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:39.442813 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:39.442870 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:39.443811 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:39.444380 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:39.445487 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:39.445682 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:39.445751 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:39.448226 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:39.448227 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:39.448228 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:39.448234 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:39.449632 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:39.450132 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:39.450133 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:39.450134 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:39.450192 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:39.450490 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:39.450490 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:39.450490 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:39.450600 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:39.450650 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:39.451000 (+ 350us) server_negotiation.cc:299] Negotiation successful
0928 21:54:39.451074 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 21:54:39.451411 11147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:39.442059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:45403 (local address 127.2.182.65:45731)
0928 21:54:39.442209 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:39.442251 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:39.442719 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:39.442931 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:39.442934 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:39.443533 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:39.443724 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:39.443729 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:39.444477 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:39.444478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:39.445375 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:39.445377 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:39.445783 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:39.445784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:39.445899 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:39.446499 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:39.446510 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:39.448118 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:39.449721 (+ 1603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:39.449731 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:39.449734 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:39.450040 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:39.450276 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:39.450276 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:39.450277 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:39.450421 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:39.450659 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:39.450664 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:39.450850 (+ 186us) client_negotiation.cc:769] Sending connection context
0928 21:54:39.450971 (+ 121us) client_negotiation.cc:240] Negotiation successful
0928 21:54:39.451147 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":77,"threads_started":1}
I20260928 21:54:39.451917 11145 heartbeater.cc:347] Connected to a master server at 127.2.182.126:45403
I20260928 21:54:39.452025 11145 heartbeater.cc:464] Registering TS with master...
I20260928 21:54:39.452222 11145 heartbeater.cc:511] Master 127.2.182.126:45403 requested a full tablet report, sending...
I20260928 21:54:39.452849 10954 ts_manager.cc:194] Registered new tserver with Master: 60b3505edffb416e9a2baef327b50117 (127.2.182.65:45765)
I20260928 21:54:39.453512 10954 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.65@KRBTEST.COM'} at 127.2.182.65:45731
WARNING: no policy specified for kudu/127.2.182.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab.
Entry for principal kudu/127.2.182.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab.
I20260928 21:54:39.475307 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.66:0
--local_ip_for_outbound_sockets=127.2.182.66
--webserver_interface=127.2.182.66
--webserver_port=0
--tserver_master_addrs=127.2.182.126:45403
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab
--principal=kudu/127.2.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:34627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:54:39.557634 11150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:54:39.557850 11150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:54:39.557883 11150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:54:39.559123 11150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:54:39.559170 11150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:54:39.559204 11150 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.66
I20260928 21:54:39.560726 11150 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:34627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab
--principal=kudu/127.2.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.182.66
--webserver_port=0
--tserver_master_addrs=127.2.182.126:45403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11150
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.66
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:54:39.561055 11150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:54:39.561297 11150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:54:39.566642 11150 server_base.cc:1034] running on GCE node
W20260928 21:54:39.566620 11156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:39.566612 11158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:39.566612 11155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:39.567106 11150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:39.567426 11150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:39.568591 11150 hybrid_clock.cc:648] HybridClock initialized: now 1790632479568578 us; error 29 us; skew 500 ppm
Sep 28 21:54:39 dist-test-slave-k6lz krb5kdc[10909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632479, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:54:39.571055 11150 init.cc:377] Logged in from keytab as kudu/127.2.182.66@KRBTEST.COM (short username kudu)
I20260928 21:54:39.571547 11150 webserver.cc:460] Webserver started at http://127.2.182.66:39429/ using document root <none> and password file <none>
I20260928 21:54:39.571745 11150 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:39.571785 11150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:39.571872 11150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:54:39.572839 11150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-1/data/instance:
uuid: "0b04b0fd371045b0b7be1c759705d97c"
format_stamp: "Formatted at 2026-09-28 21:54:39 on dist-test-slave-k6lz"
server_key: "665ce502d09d54dfea993b8d7e1b0b19"
server_key_iv: "81885e479e3251552c7fe11a0655c275"
server_key_version: "encryptionkey@0"
I20260928 21:54:39.573108 11150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-1/wal/instance:
uuid: "0b04b0fd371045b0b7be1c759705d97c"
format_stamp: "Formatted at 2026-09-28 21:54:39 on dist-test-slave-k6lz"
server_key: "665ce502d09d54dfea993b8d7e1b0b19"
server_key_iv: "81885e479e3251552c7fe11a0655c275"
server_key_version: "encryptionkey@0"
I20260928 21:54:39.574462 11150 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 21:54:39.575368 11165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:39.575588 11150 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:54:39.575677 11150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-1/wal
uuid: "0b04b0fd371045b0b7be1c759705d97c"
format_stamp: "Formatted at 2026-09-28 21:54:39 on dist-test-slave-k6lz"
server_key: "665ce502d09d54dfea993b8d7e1b0b19"
server_key_iv: "81885e479e3251552c7fe11a0655c275"
server_key_version: "encryptionkey@0"
I20260928 21:54:39.575749 11150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:39.614953 11150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:54:39.617023 11150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:54:39.617159 11150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:54:39.617378 11150 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:54:39.617662 11150 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:54:39.617697 11150 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:39.617823 11150 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:54:39.617851 11150 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:39.623126 11150 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.66:42167
I20260928 21:54:39.623203 11278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.66:42167 every 8 connection(s)
I20260928 21:54:39.623482 11150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
Sep 28 21:54:39 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632479, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.66@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:54:39.630584 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 11150
I20260928 21:54:39.630671 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-1/wal/instance
I20260928 21:54:39.630868 2777 external_mini_cluster.cc:1428] Setting key 4c76cf28fab77ef5c0b311a754312133
I20260928 21:54:39.633175 10994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:39.624167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.66:60031 (local address 127.2.182.126:45403)
0928 21:54:39.624218 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:39.624220 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:39.624433 (+ 213us) server_negotiation.cc:407] Connection header received
0928 21:54:39.624977 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:39.624978 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:39.624990 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:39.625049 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:39.626027 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:39.626566 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:39.627615 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:39.627807 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:39.627875 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:39.630086 (+ 2211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:39.630087 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:39.630087 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:39.630095 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:39.631653 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:39.632192 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:39.632192 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:39.632193 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:39.632245 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:39.632492 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:39.632493 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:39.632493 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:39.632598 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:39.632647 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:39.632988 (+ 341us) server_negotiation.cc:299] Negotiation successful
0928 21:54:39.633073 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 21:54:39.633287 11281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:39.624158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:45403 (local address 127.2.182.66:60031)
0928 21:54:39.624328 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:39.624363 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:39.624836 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:39.625151 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:39.625155 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:39.625727 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:39.625924 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:39.625929 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:39.626654 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:39.626655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:39.627552 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:39.627554 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:39.627915 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:39.627916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:39.628025 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:39.628543 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:39.628551 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:39.629988 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:39.631736 (+ 1748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:39.631744 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:39.631746 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:39.632113 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:39.632308 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:39.632308 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:39.632309 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:39.632440 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:39.632651 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:39.632656 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:39.632810 (+ 154us) client_negotiation.cc:769] Sending connection context
0928 21:54:39.632932 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 21:54:39.633103 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":71,"threads_started":1}
I20260928 21:54:39.633731 11279 heartbeater.cc:347] Connected to a master server at 127.2.182.126:45403
I20260928 21:54:39.633853 11279 heartbeater.cc:464] Registering TS with master...
I20260928 21:54:39.634028 11279 heartbeater.cc:511] Master 127.2.182.126:45403 requested a full tablet report, sending...
I20260928 21:54:39.634404 10954 ts_manager.cc:194] Registered new tserver with Master: 0b04b0fd371045b0b7be1c759705d97c (127.2.182.66:42167)
I20260928 21:54:39.634754 10954 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.66@KRBTEST.COM'} at 127.2.182.66:60031
WARNING: no policy specified for kudu/127.2.182.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab.
Entry for principal kudu/127.2.182.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab.
I20260928 21:54:39.658951 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.67:0
--local_ip_for_outbound_sockets=127.2.182.67
--webserver_interface=127.2.182.67
--webserver_port=0
--tserver_master_addrs=127.2.182.126:45403
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab
--principal=kudu/127.2.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:34627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:54:39.739264 11284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:54:39.739518 11284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:54:39.739542 11284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:54:39.740739 11284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:54:39.740785 11284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:54:39.740811 11284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.67
I20260928 21:54:39.742273 11284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:34627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab
--principal=kudu/127.2.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.182.67
--webserver_port=0
--tserver_master_addrs=127.2.182.126:45403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.67
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:54:39.742501 11284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:54:39.742707 11284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:54:39.747681 11289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:39.747741 11290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:39.747675 11292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:39.747951 11284 server_base.cc:1034] running on GCE node
I20260928 21:54:39.748134 11284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:39.748365 11284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:39.749516 11284 hybrid_clock.cc:648] HybridClock initialized: now 1790632479749500 us; error 28 us; skew 500 ppm
Sep 28 21:54:39 dist-test-slave-k6lz krb5kdc[10909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632479, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:54:39.751900 11284 init.cc:377] Logged in from keytab as kudu/127.2.182.67@KRBTEST.COM (short username kudu)
I20260928 21:54:39.752305 11284 webserver.cc:460] Webserver started at http://127.2.182.67:45657/ using document root <none> and password file <none>
I20260928 21:54:39.752508 11284 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:39.752553 11284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:39.752674 11284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:54:39.753731 11284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-2/data/instance:
uuid: "50078c19a32241688ba782be941068cc"
format_stamp: "Formatted at 2026-09-28 21:54:39 on dist-test-slave-k6lz"
server_key: "cda85dbbca762edee7aa71508e489353"
server_key_iv: "34287e1c80f32d8d043eedd151932878"
server_key_version: "encryptionkey@0"
I20260928 21:54:39.754060 11284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-2/wal/instance:
uuid: "50078c19a32241688ba782be941068cc"
format_stamp: "Formatted at 2026-09-28 21:54:39 on dist-test-slave-k6lz"
server_key: "cda85dbbca762edee7aa71508e489353"
server_key_iv: "34287e1c80f32d8d043eedd151932878"
server_key_version: "encryptionkey@0"
I20260928 21:54:39.755229 11284 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:54:39.755988 11299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:39.756150 11284 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 21:54:39.756228 11284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-2/wal
uuid: "50078c19a32241688ba782be941068cc"
format_stamp: "Formatted at 2026-09-28 21:54:39 on dist-test-slave-k6lz"
server_key: "cda85dbbca762edee7aa71508e489353"
server_key_iv: "34287e1c80f32d8d043eedd151932878"
server_key_version: "encryptionkey@0"
I20260928 21:54:39.756294 11284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:39.776608 11284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:54:39.778710 11284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:54:39.778851 11284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:54:39.779063 11284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:54:39.779367 11284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:54:39.779402 11284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:39.779429 11284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:54:39.779455 11284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:39.785091 11284 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.67:39075
I20260928 21:54:39.785151 11412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.67:39075 every 8 connection(s)
I20260928 21:54:39.785430 11284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
Sep 28 21:54:39 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632479, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.67@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:54:39.794081 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 11284
I20260928 21:54:39.794174 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-2/wal/instance
I20260928 21:54:39.794341 2777 external_mini_cluster.cc:1428] Setting key e7827791e05c04f4cd805b7aa462b979
I20260928 21:54:39.795537 10994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:39.786276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.67:51739 (local address 127.2.182.126:45403)
0928 21:54:39.786341 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:39.786343 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:39.786475 (+ 132us) server_negotiation.cc:407] Connection header received
0928 21:54:39.787096 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:39.787097 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:39.787109 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:39.787160 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:39.788188 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:39.788717 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:39.789716 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:39.789910 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:39.789964 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:39.792269 (+ 2305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:39.792271 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:39.792272 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:39.792280 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:39.793847 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:39.794397 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:39.794398 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:39.794399 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:39.794461 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:39.794764 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:39.794765 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:39.794765 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:39.794997 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:39.795065 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:39.795389 (+ 324us) server_negotiation.cc:299] Negotiation successful
0928 21:54:39.795473 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 21:54:39.795743 11415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:39.786164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:45403 (local address 127.2.182.67:51739)
0928 21:54:39.786349 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:39.786396 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:39.786991 (+ 595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:39.787230 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:39.787234 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:39.787898 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:39.788098 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:39.788103 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:39.788809 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:39.788810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:39.789636 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:39.789638 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:39.789981 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:39.789982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:39.790093 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:39.790589 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:39.790598 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:39.792174 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:39.794035 (+ 1861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:39.794044 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:39.794047 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:39.794329 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:39.794531 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:39.794532 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:39.794532 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:39.794687 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:39.795063 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:39.795068 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:39.795233 (+ 165us) client_negotiation.cc:769] Sending connection context
0928 21:54:39.795376 (+ 143us) client_negotiation.cc:240] Negotiation successful
0928 21:54:39.795534 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":76,"threads_started":1}
I20260928 21:54:39.796139 11413 heartbeater.cc:347] Connected to a master server at 127.2.182.126:45403
I20260928 21:54:39.796241 11413 heartbeater.cc:464] Registering TS with master...
I20260928 21:54:39.796383 11413 heartbeater.cc:511] Master 127.2.182.126:45403 requested a full tablet report, sending...
I20260928 21:54:39.796707 10954 ts_manager.cc:194] Registered new tserver with Master: 50078c19a32241688ba782be941068cc (127.2.182.67:39075)
I20260928 21:54:39.797087 10954 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.67@KRBTEST.COM'} at 127.2.182.67:51739
I20260928 21:54:39.806054 2777 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.2.182.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.182.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.182.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/krb5kdc/oryx_127.2.182.65.keytab.
Entry for principal oryx/127.2.182.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/krb5kdc/oryx_127.2.182.65.keytab.
WARNING: no policy specified for oryx/127.2.182.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.182.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.182.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/krb5kdc/oryx_127.2.182.66.keytab.
Entry for principal oryx/127.2.182.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/krb5kdc/oryx_127.2.182.66.keytab.
WARNING: no policy specified for oryx/127.2.182.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.182.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.182.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/krb5kdc/oryx_127.2.182.67.keytab.
Entry for principal oryx/127.2.182.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/krb5kdc/oryx_127.2.182.67.keytab.
I20260928 21:54:39.883610 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 11016
I20260928 21:54:39.887777 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 11150
I20260928 21:54:39.891665 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 11284
I20260928 21:54:39.895299 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 10922
I20260928 21:54:39.899883 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:45403
--webserver_interface=127.2.182.126
--webserver_port=42003
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab
--principal=kudu/127.2.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:34627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.126:45403
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:54:39.980217 11422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:54:39.980432 11422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:54:39.980455 11422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:54:39.981621 11422 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:54:39.981662 11422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:54:39.981675 11422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:54:39.981686 11422 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:54:39.981698 11422 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:54:39.983179 11422 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:34627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.126:45403
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab
--principal=kudu/127.2.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:45403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.126
--webserver_port=42003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11422
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:54:39.983434 11422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:54:39.983632 11422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:54:39.988612 11430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:39.988602 11428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:39.988602 11427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:39.988646 11422 server_base.cc:1034] running on GCE node
I20260928 21:54:39.989233 11422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:39.989570 11422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:39.990738 11422 hybrid_clock.cc:648] HybridClock initialized: now 1790632479990670 us; error 85 us; skew 500 ppm
Sep 28 21:54:39 dist-test-slave-k6lz krb5kdc[10909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632479, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:54:39.993361 11422 init.cc:377] Logged in from keytab as kudu/127.2.182.126@KRBTEST.COM (short username kudu)
I20260928 21:54:39.993858 11422 webserver.cc:460] Webserver started at http://127.2.182.126:42003/ using document root <none> and password file <none>
I20260928 21:54:39.994099 11422 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:39.994163 11422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:39.995584 11422 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:54:39.996172 11437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:39.996323 11422 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:39.996429 11422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/master-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/master-0/wal
uuid: "b394a9607b4c4930a4bd9a024700d54f"
format_stamp: "Formatted at 2026-09-28 21:54:39 on dist-test-slave-k6lz"
server_key: "ec2c64f12effa6cce716134ea5eeb7d0"
server_key_iv: "a4db1a76de42161dbb5bec0c7b1cf79a"
server_key_version: "encryptionkey@0"
I20260928 21:54:39.996783 11422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:40.022678 11422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:54:40.024956 11422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:54:40.025089 11422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:54:40.028803 11422 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.126:45403
I20260928 21:54:40.028855 11489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.126:45403 every 8 connection(s)
I20260928 21:54:40.029203 11422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
I20260928 21:54:40.030709 11490 sys_catalog.cc:263] Verifying existing consensus state
I20260928 21:54:40.031391 11490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b394a9607b4c4930a4bd9a024700d54f: Bootstrap starting.
I20260928 21:54:40.035097 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 11422
I20260928 21:54:40.035420 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.65:45765
--local_ip_for_outbound_sockets=127.2.182.65
--tserver_master_addrs=127.2.182.126:45403
--webserver_port=40535
--webserver_interface=127.2.182.65
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab
--principal=kudu/127.2.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:34627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.182.65
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/krb5kdc/oryx_127.2.182.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:54:40.039651 11490 log.cc:824] T 00000000000000000000000000000000 P b394a9607b4c4930a4bd9a024700d54f: Log is configured to *not* fsync() on all Append() calls
I20260928 21:54:40.041399 11490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b394a9607b4c4930a4bd9a024700d54f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 21:54:40.041658 11490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b394a9607b4c4930a4bd9a024700d54f: Bootstrap complete.
I20260928 21:54:40.043849 11490 raft_consensus.cc:348] T 00000000000000000000000000000000 P b394a9607b4c4930a4bd9a024700d54f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b394a9607b4c4930a4bd9a024700d54f" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 45403 } }
I20260928 21:54:40.044085 11490 raft_consensus.cc:729] T 00000000000000000000000000000000 P b394a9607b4c4930a4bd9a024700d54f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b394a9607b4c4930a4bd9a024700d54f, State: Initialized, Role: FOLLOWER
I20260928 21:54:40.044235 11490 consensus_queue.cc:260] T 00000000000000000000000000000000 P b394a9607b4c4930a4bd9a024700d54f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b394a9607b4c4930a4bd9a024700d54f" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 45403 } }
I20260928 21:54:40.044308 11490 raft_consensus.cc:388] T 00000000000000000000000000000000 P b394a9607b4c4930a4bd9a024700d54f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:54:40.044358 11490 raft_consensus.cc:482] T 00000000000000000000000000000000 P b394a9607b4c4930a4bd9a024700d54f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:54:40.044404 11490 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b394a9607b4c4930a4bd9a024700d54f [term 1 FOLLOWER]: Advancing to term 2
I20260928 21:54:40.045387 11490 raft_consensus.cc:504] T 00000000000000000000000000000000 P b394a9607b4c4930a4bd9a024700d54f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b394a9607b4c4930a4bd9a024700d54f" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 45403 } }
I20260928 21:54:40.045480 11490 leader_election.cc:302] T 00000000000000000000000000000000 P b394a9607b4c4930a4bd9a024700d54f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b394a9607b4c4930a4bd9a024700d54f; no voters:
I20260928 21:54:40.045658 11490 leader_election.cc:288] T 00000000000000000000000000000000 P b394a9607b4c4930a4bd9a024700d54f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 21:54:40.045706 11494 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b394a9607b4c4930a4bd9a024700d54f [term 2 FOLLOWER]: Leader election won for term 2
I20260928 21:54:40.045825 11494 raft_consensus.cc:686] T 00000000000000000000000000000000 P b394a9607b4c4930a4bd9a024700d54f [term 2 LEADER]: Becoming Leader. State: Replica: b394a9607b4c4930a4bd9a024700d54f, State: Running, Role: LEADER
I20260928 21:54:40.045928 11490 sys_catalog.cc:564] T 00000000000000000000000000000000 P b394a9607b4c4930a4bd9a024700d54f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:54:40.045912 11494 consensus_queue.cc:237] T 00000000000000000000000000000000 P b394a9607b4c4930a4bd9a024700d54f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b394a9607b4c4930a4bd9a024700d54f" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 45403 } }
I20260928 21:54:40.046227 11494 sys_catalog.cc:455] T 00000000000000000000000000000000 P b394a9607b4c4930a4bd9a024700d54f [sys.catalog]: SysCatalogTable state changed. Reason: New leader b394a9607b4c4930a4bd9a024700d54f. Latest consensus state: current_term: 2 leader_uuid: "b394a9607b4c4930a4bd9a024700d54f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b394a9607b4c4930a4bd9a024700d54f" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 45403 } } }
I20260928 21:54:40.046304 11494 sys_catalog.cc:458] T 00000000000000000000000000000000 P b394a9607b4c4930a4bd9a024700d54f [sys.catalog]: This master's current role is: LEADER
I20260928 21:54:40.046233 11495 sys_catalog.cc:455] T 00000000000000000000000000000000 P b394a9607b4c4930a4bd9a024700d54f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b394a9607b4c4930a4bd9a024700d54f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b394a9607b4c4930a4bd9a024700d54f" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 45403 } } }
I20260928 21:54:40.046512 11495 sys_catalog.cc:458] T 00000000000000000000000000000000 P b394a9607b4c4930a4bd9a024700d54f [sys.catalog]: This master's current role is: LEADER
I20260928 21:54:40.046634 11500 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:54:40.047379 11500 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:54:40.047514 11500 catalog_manager.cc:1259] Loaded cluster ID: 2c5b3234f16443fe924a95ee6ba6ce35
I20260928 21:54:40.047554 11500 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:54:40.048421 11500 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:54:40.048617 11500 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b394a9607b4c4930a4bd9a024700d54f: Loaded TSK: 0
I20260928 21:54:40.048823 11500 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 21:54:40.139230 11492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:54:40.139497 11492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:54:40.139554 11492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:54:40.140818 11492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:54:40.140865 11492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:54:40.140909 11492 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.65
I20260928 21:54:40.142390 11492 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:34627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/krb5kdc/oryx_127.2.182.65.keytab
--principal=oryx/127.2.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.65:45765
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.182.65
--webserver_port=40535
--tserver_master_addrs=127.2.182.126:45403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11492
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.65
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:54:40.142620 11492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:54:40.142827 11492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:54:40.148806 11519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:40.148723 11516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:40.148749 11517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:40.148854 11492 server_base.cc:1034] running on GCE node
I20260928 21:54:40.149183 11492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:40.149459 11492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:40.150642 11492 hybrid_clock.cc:648] HybridClock initialized: now 1790632480150627 us; error 32 us; skew 500 ppm
Sep 28 21:54:40 dist-test-slave-k6lz krb5kdc[10909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632480, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:54:40.153482 11492 init.cc:377] Logged in from keytab as oryx/127.2.182.65@KRBTEST.COM (short username oryx)
I20260928 21:54:40.154035 11492 webserver.cc:460] Webserver started at http://127.2.182.65:40535/ using document root <none> and password file <none>
I20260928 21:54:40.154258 11492 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:40.154320 11492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:40.155759 11492 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 21:54:40.156517 11526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:40.156687 11492 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:54:40.156764 11492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-0/wal
uuid: "60b3505edffb416e9a2baef327b50117"
format_stamp: "Formatted at 2026-09-28 21:54:39 on dist-test-slave-k6lz"
server_key: "57739c7bc1e2eefb73bc26a5764321c0"
server_key_iv: "d08d6fc9a3c045805a40d0674d9bdb60"
server_key_version: "encryptionkey@0"
I20260928 21:54:40.157174 11492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:40.181989 11492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:54:40.184262 11492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:54:40.184406 11492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:54:40.184636 11492 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:54:40.184980 11492 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:54:40.185016 11492 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:40.185046 11492 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:54:40.185067 11492 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:40.190639 11492 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.65:45765
I20260928 21:54:40.190948 11639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.65:45765 every 8 connection(s)
I20260928 21:54:40.191219 11492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
Sep 28 21:54:40 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.65@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:54:40 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.65@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:54:40.201072 11643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:40.192029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:45403 (local address 127.2.182.65:35379)
0928 21:54:40.192409 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:40.192454 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:40.193070 (+ 616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:40.193529 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:40.193533 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:40.194284 (+ 751us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:40.194565 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:40.194571 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:40.195813 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:40.195814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:40.197051 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:40.197054 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:40.197713 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:40.197714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:40.197849 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:40.198421 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:40.198431 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:40.200867 (+ 2436us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.126:45403: Server oryx/127.2.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":185,"threads_started":1}
I20260928 21:54:40.201094 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 11492
W20260928 21:54:40.201146 11642 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:54:40.192130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.65:35379 (local address 127.2.182.126:45403)
0928 21:54:40.192309 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:40.192328 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:40.192566 (+ 238us) server_negotiation.cc:407] Connection header received
0928 21:54:40.193282 (+ 716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:40.193288 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:40.193333 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:40.193452 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:40.194712 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:40.195720 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:40.197155 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:40.197614 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:40.197702 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:40.201005 (+ 3303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.65:35379: BlockingRecv error: recv got EOF from 127.2.182.65:35379 (error 108)
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":85,"threads_started":1}
W20260928 21:54:40.201319 11640 heartbeater.cc:644] Failed to heartbeat to 127.2.182.126:45403 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.182.126:45403: Client connection negotiation failed: client connection to 127.2.182.126:45403: Server oryx/127.2.182.126@KRBTEST.COM not found in Kerberos database
I20260928 21:54:40.201368 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.66:42167
--local_ip_for_outbound_sockets=127.2.182.66
--tserver_master_addrs=127.2.182.126:45403
--webserver_port=39429
--webserver_interface=127.2.182.66
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab
--principal=kudu/127.2.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:34627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.182.66
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/krb5kdc/oryx_127.2.182.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 21:54:40 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.65@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:54:40 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.65@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:54:40.205868 11644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:40.201525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:45403 (local address 127.2.182.65:57675)
0928 21:54:40.201650 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:40.201661 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:40.201705 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:40.201960 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:40.201961 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:40.202092 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:40.202135 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:40.202136 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:40.202758 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:40.202758 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:40.203285 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:40.203287 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:40.203657 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:40.203658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:40.203710 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:40.204114 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:40.204118 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:40.205764 (+ 1646us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.126:45403: Server oryx/127.2.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":75,"threads_started":1}
W20260928 21:54:40.205890 11642 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:54:40.201799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.65:57675 (local address 127.2.182.126:45403)
0928 21:54:40.201855 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:40.201856 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:40.201863 (+ 7us) server_negotiation.cc:407] Connection header received
0928 21:54:40.201885 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:40.201886 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:40.201896 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:40.201944 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:40.202204 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:40.202689 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:40.203370 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:40.203580 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:40.203634 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:40.205845 (+ 2211us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.65:57675: BlockingRecv error: recv got EOF from 127.2.182.65:57675 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 28 21:54:40 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.65@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:54:40 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.65@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:54:40.210812 11644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:40.206261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:45403 (local address 127.2.182.65:36231)
0928 21:54:40.206310 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:40.206322 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:40.206368 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:40.206512 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:40.206512 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:40.206653 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:40.206693 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:40.206695 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:40.207261 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:40.207262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:40.208094 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:40.208097 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:40.208468 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:40.208469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:40.208532 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:40.208990 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:40.208994 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:40.210730 (+ 1736us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.126:45403: Server oryx/127.2.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 21:54:40.210876 11642 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:54:40.206309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.65:36231 (local address 127.2.182.126:45403)
0928 21:54:40.206352 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:40.206354 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:40.206362 (+ 8us) server_negotiation.cc:407] Connection header received
0928 21:54:40.206440 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:40.206441 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:40.206449 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:40.206491 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:40.206755 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:40.207195 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:40.208181 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:40.208359 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:40.208452 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:40.210825 (+ 2373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.65:36231: BlockingRecv error: recv got EOF from 127.2.182.65:36231 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260928 21:54:40.210999 11640 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 21:54:40.281816 11645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:54:40.282052 11645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:54:40.282078 11645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:54:40.283643 11645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:54:40.283703 11645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:54:40.283739 11645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.66
I20260928 21:54:40.285377 11645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:34627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/krb5kdc/oryx_127.2.182.66.keytab
--principal=oryx/127.2.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.66:42167
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.182.66
--webserver_port=39429
--tserver_master_addrs=127.2.182.126:45403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.66
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:54:40.285646 11645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:54:40.285882 11645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:54:40.291055 11650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:40.291064 11651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:40.291136 11645 server_base.cc:1034] running on GCE node
W20260928 21:54:40.291049 11653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:40.291474 11645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:40.291707 11645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:40.292866 11645 hybrid_clock.cc:648] HybridClock initialized: now 1790632480292826 us; error 60 us; skew 500 ppm
Sep 28 21:54:40 dist-test-slave-k6lz krb5kdc[10909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632480, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:54:40.295853 11645 init.cc:377] Logged in from keytab as oryx/127.2.182.66@KRBTEST.COM (short username oryx)
I20260928 21:54:40.296347 11645 webserver.cc:460] Webserver started at http://127.2.182.66:39429/ using document root <none> and password file <none>
I20260928 21:54:40.296569 11645 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:40.296625 11645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:40.298223 11645 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 21:54:40.298990 11660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:40.299196 11645 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:54:40.299281 11645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-1/wal
uuid: "0b04b0fd371045b0b7be1c759705d97c"
format_stamp: "Formatted at 2026-09-28 21:54:39 on dist-test-slave-k6lz"
server_key: "665ce502d09d54dfea993b8d7e1b0b19"
server_key_iv: "81885e479e3251552c7fe11a0655c275"
server_key_version: "encryptionkey@0"
I20260928 21:54:40.299757 11645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:40.314021 11645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:54:40.316395 11645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:54:40.316529 11645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:54:40.316722 11645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:54:40.316984 11645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:54:40.317013 11645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:40.317035 11645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:54:40.317046 11645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:40.322436 11645 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.66:42167
I20260928 21:54:40.322476 11773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.66:42167 every 8 connection(s)
I20260928 21:54:40.322795 11645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
I20260928 21:54:40.326155 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 11645
I20260928 21:54:40.326458 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.67:39075
--local_ip_for_outbound_sockets=127.2.182.67
--tserver_master_addrs=127.2.182.126:45403
--webserver_port=45657
--webserver_interface=127.2.182.67
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab
--principal=kudu/127.2.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:34627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.182.67
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/krb5kdc/oryx_127.2.182.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 21:54:40 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.66@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:54:40 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.66@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
W20260928 21:54:40.330982 11642 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:54:40.323541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.66:58307 (local address 127.2.182.126:45403)
0928 21:54:40.323610 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:40.323611 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:40.323859 (+ 248us) server_negotiation.cc:407] Connection header received
0928 21:54:40.324367 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:40.324369 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:40.324382 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:40.324549 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:40.325591 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:40.326197 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:40.327446 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:40.327692 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:40.327751 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:40.330925 (+ 3174us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.66:58307: BlockingRecv error: recv got EOF from 127.2.182.66:58307 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 21:54:40.331043 11776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:40.323520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:45403 (local address 127.2.182.66:58307)
0928 21:54:40.323696 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:40.323739 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:40.324215 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:40.324627 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:40.324631 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:40.325268 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:40.325475 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:40.325480 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:40.326327 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:40.326328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:40.327292 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:40.327295 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:40.327775 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:40.327777 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:40.327928 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:40.328453 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:40.328463 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:40.330815 (+ 2352us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.126:45403: Server oryx/127.2.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":88,"threads_started":1}
W20260928 21:54:40.331264 11774 heartbeater.cc:644] Failed to heartbeat to 127.2.182.126:45403 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.182.126:45403: Client connection negotiation failed: client connection to 127.2.182.126:45403: Server oryx/127.2.182.126@KRBTEST.COM not found in Kerberos database
Sep 28 21:54:40 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.66@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:54:40 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.66@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:54:40.336798 11776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:40.331525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:45403 (local address 127.2.182.66:48191)
0928 21:54:40.331702 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:40.331713 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:40.331776 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:40.331912 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:40.331913 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:40.332063 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:40.332124 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:40.332126 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:40.332908 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:40.332909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:40.333730 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:40.333734 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:40.334241 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:40.334242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:40.334314 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:40.334830 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:40.334835 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:40.336724 (+ 1889us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.126:45403: Server oryx/127.2.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153}
W20260928 21:54:40.336874 11642 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:54:40.331641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.66:48191 (local address 127.2.182.126:45403)
0928 21:54:40.331709 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:40.331712 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:40.331756 (+ 44us) server_negotiation.cc:407] Connection header received
0928 21:54:40.331836 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:40.331837 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:40.331845 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:40.331892 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:40.332247 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:40.332735 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:40.333806 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:40.334166 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:40.334225 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:40.336818 (+ 2593us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.66:48191: BlockingRecv error: recv got EOF from 127.2.182.66:48191 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 28 21:54:40 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.66@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:54:40 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.66@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:54:40.341249 11776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:40.337102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:45403 (local address 127.2.182.66:56049)
0928 21:54:40.337147 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:40.337156 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:40.337202 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:40.337317 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:40.337318 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:40.337443 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:40.337474 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:40.337475 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:40.338066 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:40.338067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:40.338561 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:40.338563 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:40.338874 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:40.338875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:40.338926 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:40.339433 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:40.339438 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:40.341192 (+ 1754us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.126:45403: Server oryx/127.2.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 21:54:40.341375 11642 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:54:40.337156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.66:56049 (local address 127.2.182.126:45403)
0928 21:54:40.337196 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:40.337198 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:40.337205 (+ 7us) server_negotiation.cc:407] Connection header received
0928 21:54:40.337260 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:40.337261 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:40.337270 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:40.337309 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:40.337535 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:40.338003 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:40.338628 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:40.338808 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:40.338863 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:40.341309 (+ 2446us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.66:56049: BlockingRecv error: recv got EOF from 127.2.182.66:56049 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 21:54:40.341372 11774 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 21:54:40.410719 11777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:54:40.410951 11777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:54:40.410975 11777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:54:40.412207 11777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:54:40.412261 11777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:54:40.412289 11777 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.67
I20260928 21:54:40.413789 11777 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:34627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/krb5kdc/oryx_127.2.182.67.keytab
--principal=oryx/127.2.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.67:39075
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.182.67
--webserver_port=45657
--tserver_master_addrs=127.2.182.126:45403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11777
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.67
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:54:40.414034 11777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:54:40.414247 11777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:54:40.419502 11785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:40.419510 11783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:40.419541 11782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:40.419552 11777 server_base.cc:1034] running on GCE node
I20260928 21:54:40.419878 11777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:40.420161 11777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:40.421305 11777 hybrid_clock.cc:648] HybridClock initialized: now 1790632480421289 us; error 25 us; skew 500 ppm
Sep 28 21:54:40 dist-test-slave-k6lz krb5kdc[10909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632480, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:54:40.423622 11777 init.cc:377] Logged in from keytab as oryx/127.2.182.67@KRBTEST.COM (short username oryx)
I20260928 21:54:40.424047 11777 webserver.cc:460] Webserver started at http://127.2.182.67:45657/ using document root <none> and password file <none>
I20260928 21:54:40.424240 11777 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:40.424289 11777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:40.425595 11777 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:54:40.426146 11792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:40.426326 11777 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:40.426406 11777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-2/wal
uuid: "50078c19a32241688ba782be941068cc"
format_stamp: "Formatted at 2026-09-28 21:54:39 on dist-test-slave-k6lz"
server_key: "cda85dbbca762edee7aa71508e489353"
server_key_iv: "34287e1c80f32d8d043eedd151932878"
server_key_version: "encryptionkey@0"
I20260928 21:54:40.426734 11777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:40.439965 11777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:54:40.442162 11777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:54:40.442276 11777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:54:40.442487 11777 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:54:40.442773 11777 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:54:40.442806 11777 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:40.442835 11777 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:54:40.442847 11777 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:40.448431 11777 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.67:39075
I20260928 21:54:40.448474 11905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.67:39075 every 8 connection(s)
I20260928 21:54:40.448807 11777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
I20260928 21:54:40.451736 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 11777
Sep 28 21:54:40 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.67@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:54:40 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.67@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
W20260928 21:54:40.456854 11642 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:54:40.449642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.67:37737 (local address 127.2.182.126:45403)
0928 21:54:40.449696 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:40.449698 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:40.449935 (+ 237us) server_negotiation.cc:407] Connection header received
0928 21:54:40.450491 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:40.450492 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:40.450505 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:40.450555 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:40.451548 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:40.452161 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:40.453295 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:40.453528 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:40.453590 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:40.456786 (+ 3196us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.67:37737: BlockingRecv error: recv got EOF from 127.2.182.67:37737 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 21:54:40.456951 11908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:40.449626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:45403 (local address 127.2.182.67:37737)
0928 21:54:40.449791 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:40.449835 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:40.450376 (+ 541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:40.450634 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:40.450638 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:40.451209 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:40.451457 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:40.451462 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:40.452254 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:40.452255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:40.453195 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:40.453197 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:40.453613 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:40.453614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:40.453778 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:40.454279 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:40.454287 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:40.456715 (+ 2428us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.126:45403: Server oryx/127.2.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":78,"threads_started":1}
W20260928 21:54:40.457275 11906 heartbeater.cc:644] Failed to heartbeat to 127.2.182.126:45403 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.182.126:45403: Client connection negotiation failed: client connection to 127.2.182.126:45403: Server oryx/127.2.182.126@KRBTEST.COM not found in Kerberos database
I20260928 21:54:40.459296 11910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:40.452224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42970 (local address 127.2.182.126:45403)
0928 21:54:40.452344 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:40.452346 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:40.452450 (+ 104us) server_negotiation.cc:407] Connection header received
0928 21:54:40.452497 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:40.452497 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:40.452508 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:40.452568 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:40.453125 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:40.453717 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:40.454493 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:40.454789 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:40.454855 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:40.456115 (+ 1260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:40.456120 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:40.456120 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:40.456131 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:40.457882 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:40.458203 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:40.458206 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:40.458206 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:40.458323 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:40.458498 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:40.458499 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:40.458499 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:40.458755 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:40.458853 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:40.459090 (+ 237us) server_negotiation.cc:299] Negotiation successful
0928 21:54:40.459205 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":102,"thread_start_us":56,"threads_started":1}
Sep 28 21:54:40 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.67@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:54:40 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.67@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:54:40.461856 11908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:40.457548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:45403 (local address 127.2.182.67:45631)
0928 21:54:40.457589 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:40.457600 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:40.457638 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:40.457742 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:40.457743 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:40.457867 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:40.457911 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:40.457913 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:40.458680 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:40.458680 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:40.459433 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:40.459436 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:40.459843 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:40.459844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:40.459971 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:40.460356 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:40.460360 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:40.461808 (+ 1448us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.126:45403: Server oryx/127.2.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 21:54:40.461962 11642 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:54:40.457599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.67:45631 (local address 127.2.182.126:45403)
0928 21:54:40.457638 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:40.457640 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:40.457648 (+ 8us) server_negotiation.cc:407] Connection header received
0928 21:54:40.457687 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:40.457688 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:40.457697 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:40.457740 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:40.457983 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:40.458610 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:40.459516 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:40.459774 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:40.459832 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:40.461907 (+ 2075us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.67:45631: BlockingRecv error: recv got EOF from 127.2.182.67:45631 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
Sep 28 21:54:40 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.67@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:54:40 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.67@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:54:40.465958 11908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:40.462121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:45403 (local address 127.2.182.67:37347)
0928 21:54:40.462161 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:40.462170 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:40.462201 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:40.462357 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:40.462358 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:40.462456 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:40.462490 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:40.462491 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:40.463042 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:40.463042 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:40.463582 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:40.463584 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:40.463933 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:40.463934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:40.463981 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:40.464337 (+ 356us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:40.464341 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:40.465909 (+ 1568us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.126:45403: Server oryx/127.2.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":23}
W20260928 21:54:40.466073 11642 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:54:40.462181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.67:37347 (local address 127.2.182.126:45403)
0928 21:54:40.462220 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:40.462222 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:40.462229 (+ 7us) server_negotiation.cc:407] Connection header received
0928 21:54:40.462248 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:40.462248 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:40.462256 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:40.462301 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:40.462538 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:40.462979 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:40.463657 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:40.463833 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:40.463897 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:40.466016 (+ 2119us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.67:37347: BlockingRecv error: recv got EOF from 127.2.182.67:37347 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260928 21:54:40.466122 11906 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 28 21:54:41 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.65@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:54:41 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.65@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:54:41.217084 11911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:41.211713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:45403 (local address 127.2.182.65:52037)
0928 21:54:41.211865 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:41.211878 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:41.211946 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:41.212110 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:41.212111 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:41.212268 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:41.212316 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:41.212318 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:41.213147 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:41.213148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:41.213697 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:41.213699 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:41.214323 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:41.214324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:41.214382 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:41.214942 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:41.214946 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:41.216990 (+ 2044us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.126:45403: Server oryx/127.2.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":77,"threads_started":1}
W20260928 21:54:41.217157 11912 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:54:41.211832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.65:52037 (local address 127.2.182.126:45403)
0928 21:54:41.211944 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:41.211946 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:41.211956 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:54:41.211997 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:41.211998 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:41.212008 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:41.212067 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:41.212471 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:41.213024 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:41.213907 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:41.214200 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:41.214269 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:41.217097 (+ 2828us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.65:52037: BlockingRecv error: recv got EOF from 127.2.182.65:52037 (error 108)
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":48,"threads_started":1}
Sep 28 21:54:41 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.66@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:54:41 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.66@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:54:41.346490 11913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:41.341777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:45403 (local address 127.2.182.66:54705)
0928 21:54:41.341899 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:41.341911 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:41.341971 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:41.342104 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:41.342104 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:41.342277 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:41.342325 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:41.342327 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:41.342989 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:41.342990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:41.343586 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:41.343589 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:41.343915 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:41.343916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:41.343969 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:41.344432 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:41.344436 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:41.346325 (+ 1889us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.126:45403: Server oryx/127.2.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":54,"threads_started":1}
W20260928 21:54:41.346561 11912 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:54:41.341877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.66:54705 (local address 127.2.182.126:45403)
0928 21:54:41.341948 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:41.341950 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:41.341960 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:54:41.342033 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:41.342034 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:41.342048 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:41.342089 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:41.342404 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:41.342908 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:41.343658 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:41.343851 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:41.343913 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:41.346501 (+ 2588us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.66:54705: BlockingRecv error: recv got EOF from 127.2.182.66:54705 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 28 21:54:41 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.67@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:54:41 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.67@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:54:41.471391 11914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:41.466606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:45403 (local address 127.2.182.67:48045)
0928 21:54:41.466744 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:41.466754 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:41.466814 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:41.466999 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:41.466999 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:41.467154 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:41.467205 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:41.467207 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:41.467857 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:41.467857 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:41.468449 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:41.468451 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:41.468802 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:41.468802 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:41.468839 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:41.469277 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:41.469282 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:41.471275 (+ 1993us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.126:45403: Server oryx/127.2.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":58,"threads_started":1}
W20260928 21:54:41.471547 11912 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:54:41.466733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.67:48045 (local address 127.2.182.126:45403)
0928 21:54:41.466783 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:41.466791 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:41.466802 (+ 11us) server_negotiation.cc:407] Connection header received
0928 21:54:41.466860 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:41.466860 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:41.466872 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:41.466980 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:41.467272 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:41.467785 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:41.468544 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:41.468739 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:41.468800 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:41.471484 (+ 2684us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.67:48045: BlockingRecv error: recv got EOF from 127.2.182.67:48045 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 28 21:54:42 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.65@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:54:42 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.65@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:54:42.222692 11915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:42.217806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:45403 (local address 127.2.182.65:58739)
0928 21:54:42.217941 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:42.217953 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:42.218024 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:42.218224 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:42.218224 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:42.218421 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:42.218472 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:42.218475 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:42.219211 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:42.219212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:42.219800 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:42.219803 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:42.220231 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:42.220232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:42.220311 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:42.220818 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:42.220823 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:42.222629 (+ 1806us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.126:45403: Server oryx/127.2.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":65,"threads_started":1}
W20260928 21:54:42.222873 11916 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:54:42.217958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.65:58739 (local address 127.2.182.126:45403)
0928 21:54:42.218076 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:42.218079 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:42.218086 (+ 7us) server_negotiation.cc:407] Connection header received
0928 21:54:42.218109 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:42.218109 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:42.218122 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:42.218205 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:42.218572 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:42.219118 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:42.219899 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:42.220130 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:42.220195 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:42.222798 (+ 2603us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.65:58739: BlockingRecv error: recv got EOF from 127.2.182.65:58739 (error 108)
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":65,"threads_started":1}
Sep 28 21:54:42 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.66@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:54:42 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.66@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:54:42.352737 11917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:42.346993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:45403 (local address 127.2.182.66:45709)
0928 21:54:42.347135 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:42.347147 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:42.347203 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:42.347481 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:42.347482 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:42.347640 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:42.347689 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:42.347691 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:42.348441 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:42.348443 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:42.349012 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:42.349015 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:42.349464 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:42.349466 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:42.349521 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:42.350461 (+ 940us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:42.350467 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:42.352672 (+ 2205us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.126:45403: Server oryx/127.2.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":63,"threads_started":1}
W20260928 21:54:42.353055 11916 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:54:42.347130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.66:45709 (local address 127.2.182.126:45403)
0928 21:54:42.347177 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:42.347179 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:42.347191 (+ 12us) server_negotiation.cc:407] Connection header received
0928 21:54:42.347393 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:42.347394 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:42.347410 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:42.347464 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:42.347780 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:42.348306 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:42.349120 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:42.349320 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:42.349463 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:42.352959 (+ 3496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.66:45709: BlockingRecv error: recv got EOF from 127.2.182.66:45709 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 28 21:54:42 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.67@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:54:42 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.67@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
W20260928 21:54:42.478137 11916 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:54:42.472062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.67:38137 (local address 127.2.182.126:45403)
0928 21:54:42.472174 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:42.472176 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:42.472427 (+ 251us) server_negotiation.cc:407] Connection header received
0928 21:54:42.472487 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:42.472488 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:42.472650 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:42.472717 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:42.473185 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:42.473755 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:42.474824 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:42.475098 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:42.475167 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:42.478075 (+ 2908us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.67:38137: BlockingRecv error: recv got EOF from 127.2.182.67:38137 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 21:54:42.478075 11918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:42.471827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:45403 (local address 127.2.182.67:38137)
0928 21:54:42.472078 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:42.472097 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:42.472436 (+ 339us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:42.472760 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:42.472761 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:42.472938 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:42.473042 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:42.473045 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:42.473848 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:42.473849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:42.474657 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:42.474665 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:42.475210 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:42.475211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:42.475274 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:42.475793 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:42.475798 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:42.477902 (+ 2104us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.126:45403: Server oryx/127.2.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":66,"threads_started":1}
Sep 28 21:54:43 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.65@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:54:43 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.65@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:54:43.227926 11919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:43.223249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:45403 (local address 127.2.182.65:34803)
0928 21:54:43.223419 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:43.223430 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:43.223483 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:43.223677 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:43.223678 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:43.223835 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:43.223889 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:43.223890 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:43.224616 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:43.224618 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:43.225170 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:43.225172 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:43.225526 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:43.225527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:43.225570 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:43.226026 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:43.226031 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:43.227843 (+ 1812us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.126:45403: Server oryx/127.2.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":60,"threads_started":1}
W20260928 21:54:43.227973 11920 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:54:43.223432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.65:34803 (local address 127.2.182.126:45403)
0928 21:54:43.223544 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:43.223546 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:43.223554 (+ 8us) server_negotiation.cc:407] Connection header received
0928 21:54:43.223579 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:43.223580 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:43.223591 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:43.223661 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:43.223977 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:43.224504 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:43.225256 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:43.225462 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:43.225516 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:43.227915 (+ 2399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.65:34803: BlockingRecv error: recv got EOF from 127.2.182.65:34803 (error 108)
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":53,"threads_started":1}
Sep 28 21:54:43 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.66@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:54:43 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.66@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:54:43.358376 11921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:43.353349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:45403 (local address 127.2.182.66:49337)
0928 21:54:43.353563 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:43.353574 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:43.353643 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:43.353792 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:43.353793 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:43.353958 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:43.354010 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:43.354012 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:43.354672 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:43.354673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:43.355224 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:43.355226 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:43.355723 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:43.355724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:43.355775 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:43.356313 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:43.356319 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:43.358311 (+ 1992us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.126:45403: Server oryx/127.2.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":111,"threads_started":1}
W20260928 21:54:43.358521 11920 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:54:43.353499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.66:49337 (local address 127.2.182.126:45403)
0928 21:54:43.353563 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:43.353565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:43.353647 (+ 82us) server_negotiation.cc:407] Connection header received
0928 21:54:43.353700 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:43.353700 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:43.353711 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:43.353777 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:43.354087 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:43.354599 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:43.355433 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:43.355631 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:43.355699 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:43.358449 (+ 2750us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.66:49337: BlockingRecv error: recv got EOF from 127.2.182.66:49337 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 28 21:54:43 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.67@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:54:43 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.67@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:54:43.483958 11922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:43.478895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:45403 (local address 127.2.182.67:58569)
0928 21:54:43.479023 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:43.479033 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:43.479084 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:43.479274 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:43.479275 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:43.479499 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:43.479548 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:43.479550 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:43.480224 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:43.480225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:43.480911 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:43.480914 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:43.481308 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:43.481309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:43.481355 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:43.481862 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:43.481866 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:43.483878 (+ 2012us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.126:45403: Server oryx/127.2.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":68,"threads_started":1}
W20260928 21:54:43.484076 11920 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:54:43.479010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.67:58569 (local address 127.2.182.126:45403)
0928 21:54:43.479103 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:43.479105 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:43.479115 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:54:43.479141 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:43.479142 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:43.479154 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:43.479221 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:43.479627 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:43.480137 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:43.481029 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:43.481223 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:43.481284 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:43.484005 (+ 2721us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.67:58569: BlockingRecv error: recv got EOF from 127.2.182.67:58569 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 28 21:54:44 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.65@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:54:44 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.65@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:54:44.237979 11923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:44.228543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:45403 (local address 127.2.182.65:49069)
0928 21:54:44.228691 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:44.228702 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:44.228754 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:44.228921 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:44.228921 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:44.229084 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:44.229138 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:44.229140 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:44.229925 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:44.229926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:44.230457 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:44.230459 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:44.230781 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:44.230782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:44.230823 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:44.235967 (+ 5144us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:44.235972 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:44.237898 (+ 1926us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.126:45403: Server oryx/127.2.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":66,"threads_started":1}
W20260928 21:54:44.238060 11924 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:54:44.228661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.65:49069 (local address 127.2.182.126:45403)
0928 21:54:44.228792 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:44.228794 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:44.228803 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:54:44.228829 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:44.228830 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:44.228840 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:44.228906 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:44.229207 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:44.229771 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:44.230531 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:44.230724 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:44.230778 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:44.238001 (+ 7223us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.65:49069: BlockingRecv error: recv got EOF from 127.2.182.65:49069 (error 108)
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":64,"threads_started":1}
Sep 28 21:54:44 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.66@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:54:44 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.66@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:54:44.363812 11925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:44.358888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:45403 (local address 127.2.182.66:52215)
0928 21:54:44.359030 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:44.359043 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:44.359120 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:44.359239 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:44.359240 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:44.359470 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:44.359518 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:44.359520 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:44.360209 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:44.360210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:44.360849 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:44.360852 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:44.361254 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:44.361255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:44.361315 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:44.361786 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:44.361791 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:44.363749 (+ 1958us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.126:45403: Server oryx/127.2.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":62,"threads_started":1}
W20260928 21:54:44.364017 11924 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:54:44.359026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.66:52215 (local address 127.2.182.126:45403)
0928 21:54:44.359080 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:44.359081 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:44.359095 (+ 14us) server_negotiation.cc:407] Connection header received
0928 21:54:44.359167 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:44.359168 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:44.359178 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:44.359224 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:44.359575 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:44.360125 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:44.360952 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:44.361144 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:44.361207 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:44.363868 (+ 2661us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.66:52215: BlockingRecv error: recv got EOF from 127.2.182.66:52215 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Sep 28 21:54:44 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.67@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:54:44 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.67@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:54:44.488994 11926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:44.484548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:45403 (local address 127.2.182.67:52081)
0928 21:54:44.484694 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:44.484706 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:44.484780 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:44.485024 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:44.485025 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:44.485191 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:44.485239 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:44.485241 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:44.485877 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:44.485878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:44.486438 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:44.486440 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:44.486764 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:44.486765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:44.486816 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:44.487259 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:44.487264 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:44.488916 (+ 1652us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.126:45403: Server oryx/127.2.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":60,"threads_started":1}
W20260928 21:54:44.489075 11924 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:54:44.484689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.67:52081 (local address 127.2.182.126:45403)
0928 21:54:44.484751 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:44.484753 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:44.484762 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:54:44.484932 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:44.484933 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:44.484948 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:44.484997 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:44.485315 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:44.485805 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:44.486521 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:44.486703 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:44.486758 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:44.489020 (+ 2262us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.67:52081: BlockingRecv error: recv got EOF from 127.2.182.67:52081 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 28 21:54:45 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.65@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:54:45 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.65@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:54:45.243268 11927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:45.238574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:45403 (local address 127.2.182.65:47381)
0928 21:54:45.238712 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:45.238723 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:45.238779 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:45.238921 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:45.238922 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:45.239079 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:45.239131 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:45.239132 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:45.239896 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:45.239898 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:45.240467 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:45.240469 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:45.240853 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:45.240853 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:45.240907 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:45.241428 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:45.241433 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:45.243191 (+ 1758us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.126:45403: Server oryx/127.2.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":61,"threads_started":1}
W20260928 21:54:45.243492 11928 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:54:45.238679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.65:47381 (local address 127.2.182.126:45403)
0928 21:54:45.238792 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:45.238794 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:45.238802 (+ 8us) server_negotiation.cc:407] Connection header received
0928 21:54:45.238825 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:45.238826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:45.238836 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:45.238903 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:45.239188 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:45.239779 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:45.240563 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:45.240767 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:45.240837 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:45.243423 (+ 2586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.65:47381: BlockingRecv error: recv got EOF from 127.2.182.65:47381 (error 108)
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":60,"threads_started":1}
Sep 28 21:54:45 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.66@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:54:45 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.66@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:54:45.369481 11929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:45.364463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:45403 (local address 127.2.182.66:35917)
0928 21:54:45.364612 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:45.364625 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:45.364687 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:45.364854 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:45.364855 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:45.365027 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:45.365075 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:45.365077 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:45.365859 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:45.365859 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:45.366500 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:45.366503 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:45.366841 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:45.366842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:45.366894 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:45.367545 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:45.367550 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:45.369417 (+ 1867us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.126:45403: Server oryx/127.2.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":59,"threads_started":1}
W20260928 21:54:45.369593 11928 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:54:45.364637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.66:35917 (local address 127.2.182.126:45403)
0928 21:54:45.364696 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:45.364698 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:45.364707 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:54:45.364736 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:45.364737 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:45.364748 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:45.364808 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:45.365149 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:45.365746 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:45.366576 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:45.366774 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:45.366828 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:45.369531 (+ 2703us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.66:35917: BlockingRecv error: recv got EOF from 127.2.182.66:35917 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 28 21:54:45 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.67@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:54:45 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.67@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:54:45.494328 11930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:45.489524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:45403 (local address 127.2.182.67:42923)
0928 21:54:45.489668 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:45.489682 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:45.489824 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:45.490008 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:45.490009 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:45.490166 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:45.490214 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:45.490215 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:45.490907 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:45.490908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:45.491501 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:45.491503 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:45.491839 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:45.491840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:45.491880 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:45.492324 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:45.492329 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:45.494269 (+ 1940us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.126:45403: Server oryx/127.2.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":82,"threads_started":1}
W20260928 21:54:45.494496 11928 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:54:45.489602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.67:42923 (local address 127.2.182.126:45403)
0928 21:54:45.489673 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:45.489675 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:45.489725 (+ 50us) server_negotiation.cc:407] Connection header received
0928 21:54:45.489922 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:45.489923 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:45.489939 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:45.489997 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:45.490321 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:45.490824 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:45.491569 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:45.491751 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:45.491802 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:45.494396 (+ 2594us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.67:42923: BlockingRecv error: recv got EOF from 127.2.182.67:42923 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 28 21:54:46 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.65@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:54:46 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.65@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:54:46.249655 11931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:46.243821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:45403 (local address 127.2.182.65:38429)
0928 21:54:46.244002 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:46.244016 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:46.244088 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:46.244248 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:46.244249 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:46.244463 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:46.244525 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:46.244528 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:46.245386 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:46.245387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:46.246014 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:46.246017 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:46.246422 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:46.246423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:46.246529 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:46.247367 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:46.247372 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:46.249590 (+ 2218us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.126:45403: Server oryx/127.2.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":80,"threads_started":1}
W20260928 21:54:46.249878 11932 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:54:46.243946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.65:38429 (local address 127.2.182.126:45403)
0928 21:54:46.244110 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:46.244112 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:46.244122 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:54:46.244148 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:46.244148 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:46.244161 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:46.244235 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:46.244619 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:46.245217 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:46.246118 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:46.246329 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:46.246398 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:46.249737 (+ 3339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.65:38429: BlockingRecv error: recv got EOF from 127.2.182.65:38429 (error 108)
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":92,"threads_started":1}
Sep 28 21:54:46 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.66@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:54:46 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.66@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:54:46.374833 11933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:46.370054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:45403 (local address 127.2.182.66:38459)
0928 21:54:46.370216 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:46.370230 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:46.370369 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:46.370511 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:46.370512 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:46.370687 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:46.370734 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:46.370736 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:46.371396 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:46.371397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:46.371925 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:46.371927 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:46.372254 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:46.372255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:46.372309 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:46.372741 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:46.372746 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:46.374753 (+ 2007us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.126:45403: Server oryx/127.2.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":90,"threads_started":1}
W20260928 21:54:46.374914 11932 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:54:46.370156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.66:38459 (local address 127.2.182.126:45403)
0928 21:54:46.370235 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:46.370237 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:46.370361 (+ 124us) server_negotiation.cc:407] Connection header received
0928 21:54:46.370442 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:46.370443 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:46.370455 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:46.370503 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:46.370806 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:46.371297 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:46.372001 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:46.372186 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:46.372241 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:46.374861 (+ 2620us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.66:38459: BlockingRecv error: recv got EOF from 127.2.182.66:38459 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 28 21:54:46 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.67@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:54:46 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.67@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:54:46.499738 11934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:46.494890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:45403 (local address 127.2.182.67:49805)
0928 21:54:46.495037 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:46.495049 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:46.495136 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:46.495272 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:46.495273 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:46.495521 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:46.495569 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:46.495570 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:46.496238 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:46.496239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:46.496791 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:46.496793 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:46.497193 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:46.497201 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:46.497255 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:46.497717 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:46.497721 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:46.499660 (+ 1939us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.126:45403: Server oryx/127.2.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":85,"threads_started":1}
W20260928 21:54:46.499809 11932 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:54:46.494994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.67:49805 (local address 127.2.182.126:45403)
0928 21:54:46.495076 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:46.495078 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:46.495103 (+ 25us) server_negotiation.cc:407] Connection header received
0928 21:54:46.495199 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:46.495200 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:46.495213 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:46.495259 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:46.495655 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:46.496156 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:46.496943 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:46.497128 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:46.497184 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:46.499756 (+ 2572us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.67:49805: BlockingRecv error: recv got EOF from 127.2.182.67:49805 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 28 21:54:47 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.65@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:54:47 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.65@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:54:47.254745 11935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:47.250207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:45403 (local address 127.2.182.65:38533)
0928 21:54:47.250327 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:47.250337 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:47.250388 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:47.250567 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:47.250568 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:47.250719 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:47.250769 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:47.250770 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:47.251497 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:47.251498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:47.252028 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:47.252030 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:47.252389 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:47.252390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:47.252438 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:47.252856 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:47.252860 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:47.254664 (+ 1804us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.126:45403: Server oryx/127.2.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":68,"threads_started":1}
W20260928 21:54:47.254817 11936 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:54:47.250293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.65:38533 (local address 127.2.182.126:45403)
0928 21:54:47.250430 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:47.250431 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:47.250443 (+ 12us) server_negotiation.cc:407] Connection header received
0928 21:54:47.250467 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:47.250468 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:47.250478 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:47.250554 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:47.250860 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:47.251401 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:47.252121 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:47.252319 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:47.252373 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:47.254750 (+ 2377us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.65:38533: BlockingRecv error: recv got EOF from 127.2.182.65:38533 (error 108)
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":67,"threads_started":1}
Sep 28 21:54:47 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.66@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:54:47 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.66@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:54:47.380429 11937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:47.375478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:45403 (local address 127.2.182.66:53521)
0928 21:54:47.375672 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:47.375684 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:47.375748 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:47.375895 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:47.375896 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:47.376063 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:47.376111 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:47.376113 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:47.376911 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:47.376911 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:47.377500 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:47.377502 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:47.377894 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:47.377894 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:47.377930 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:47.378383 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:47.378387 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:47.380369 (+ 1982us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.126:45403: Server oryx/127.2.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":67,"threads_started":1}
W20260928 21:54:47.380563 11936 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:54:47.375527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.66:53521 (local address 127.2.182.126:45403)
0928 21:54:47.375591 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:47.375593 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:47.375745 (+ 152us) server_negotiation.cc:407] Connection header received
0928 21:54:47.375798 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:47.375799 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:47.375811 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:47.375864 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:47.376229 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:47.376831 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:47.377583 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:47.377832 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:47.377890 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:47.380495 (+ 2605us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.66:53521: BlockingRecv error: recv got EOF from 127.2.182.66:53521 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 28 21:54:47 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.67@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:54:47 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.67@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:54:47.504570 11938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:47.500180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:45403 (local address 127.2.182.67:40425)
0928 21:54:47.500335 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:47.500347 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:47.500413 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:47.500558 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:47.500558 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:47.500715 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:47.500770 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:47.500771 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:47.501416 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:47.501417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:47.502028 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:47.502031 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:47.502356 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:47.502357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:47.502398 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:47.502802 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:47.502807 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:47.504503 (+ 1696us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.126:45403: Server oryx/127.2.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":63,"threads_started":1}
W20260928 21:54:47.504709 11936 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:54:47.500264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.67:40425 (local address 127.2.182.126:45403)
0928 21:54:47.500328 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:47.500330 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:47.500406 (+ 76us) server_negotiation.cc:407] Connection header received
0928 21:54:47.500482 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:47.500483 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:47.500495 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:47.500542 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:47.500822 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:47.501307 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:47.502091 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:47.502291 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:47.502340 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:47.504635 (+ 2295us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.67:40425: BlockingRecv error: recv got EOF from 127.2.182.67:40425 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 28 21:54:48 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.65@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:54:48 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.65@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:54:48.260154 11939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:48.255357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:45403 (local address 127.2.182.65:53411)
0928 21:54:48.255514 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:48.255527 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:48.255591 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:48.255758 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:48.255759 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:48.256076 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:48.256137 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:48.256139 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:48.256839 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:48.256840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:48.257427 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:48.257430 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:48.257807 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:48.257809 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:48.257929 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:48.258387 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:48.258391 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:48.260095 (+ 1704us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.126:45403: Server oryx/127.2.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":57,"threads_started":1}
W20260928 21:54:48.260263 11940 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:54:48.255506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.65:53411 (local address 127.2.182.126:45403)
0928 21:54:48.255614 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:48.255615 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:48.255626 (+ 11us) server_negotiation.cc:407] Connection header received
0928 21:54:48.255654 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:48.255655 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:48.255668 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:48.255756 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:48.256231 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:48.256758 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:48.257515 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:48.257714 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:48.257771 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:48.260196 (+ 2425us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.65:53411: BlockingRecv error: recv got EOF from 127.2.182.65:53411 (error 108)
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":48,"threads_started":1}
Sep 28 21:54:48 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.66@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:54:48 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.66@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:54:48.385986 11941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:48.380916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:45403 (local address 127.2.182.66:47683)
0928 21:54:48.381091 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:48.381102 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:48.381157 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:48.381270 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:48.381271 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:48.381428 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:48.381517 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:48.381519 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:48.382487 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:48.382488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:48.383031 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:48.383033 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:48.383443 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:48.383444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:48.383503 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:48.384053 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:48.384057 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:48.385915 (+ 1858us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.126:45403: Server oryx/127.2.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":109,"threads_started":1}
W20260928 21:54:48.386114 11940 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:54:48.381079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.66:47683 (local address 127.2.182.126:45403)
0928 21:54:48.381145 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:48.381147 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:48.381156 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:54:48.381202 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:48.381202 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:48.381212 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:48.381259 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:48.381607 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:48.382361 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:48.383122 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:48.383356 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:48.383414 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:48.386047 (+ 2633us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.66:47683: BlockingRecv error: recv got EOF from 127.2.182.66:47683 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 28 21:54:48 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.67@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:54:48 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.67@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:54:48.510077 11942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:48.505089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:45403 (local address 127.2.182.67:50347)
0928 21:54:48.505220 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:48.505232 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:48.505293 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:48.505477 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:48.505477 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:48.505654 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:48.505711 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:48.505713 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:48.506401 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:48.506402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:48.506931 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:48.506933 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:48.507256 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:48.507257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:48.507305 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:48.507863 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:48.507867 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:48.510007 (+ 2140us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.126:45403: Server oryx/127.2.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":58,"threads_started":1}
W20260928 21:54:48.510190 11940 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:54:48.505232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.67:50347 (local address 127.2.182.126:45403)
0928 21:54:48.505364 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:48.505366 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:48.505376 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:54:48.505399 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:48.505399 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:48.505412 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:48.505457 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:48.505787 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:48.506322 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:48.507012 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:48.507197 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:48.507253 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:48.510126 (+ 2873us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.67:50347: BlockingRecv error: recv got EOF from 127.2.182.67:50347 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Sep 28 21:54:49 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.65@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:54:49 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.65@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:54:49.270284 11943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:49.260639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:45403 (local address 127.2.182.65:51105)
0928 21:54:49.260813 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:49.260825 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:49.260879 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:49.261053 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:49.261054 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:49.261212 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:49.261261 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:49.261262 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:49.262031 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:49.262031 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:49.262557 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:49.262558 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:49.263062 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:49.263063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:49.263138 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:49.267983 (+ 4845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:49.267989 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:49.270208 (+ 2219us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.126:45403: Server oryx/127.2.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":101,"threads_started":1}
W20260928 21:54:49.270458 11944 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:54:49.260745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.65:51105 (local address 127.2.182.126:45403)
0928 21:54:49.260874 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:49.260876 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:49.260887 (+ 11us) server_negotiation.cc:407] Connection header received
0928 21:54:49.260915 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:49.260916 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:49.260928 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:49.261008 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:49.261377 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:49.261921 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:49.262676 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:49.262966 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:49.263037 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:49.270386 (+ 7349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.65:51105: BlockingRecv error: recv got EOF from 127.2.182.65:51105 (error 108)
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":71,"threads_started":1}
Sep 28 21:54:49 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.66@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:54:49 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.66@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:54:49.391070 11945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:49.386507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:45403 (local address 127.2.182.66:58845)
0928 21:54:49.386698 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:49.386708 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:49.386755 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:49.386864 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:49.386864 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:49.387024 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:49.387074 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:49.387075 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:49.387819 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:49.387819 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:49.388360 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:49.388362 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:49.388736 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:49.388737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:49.388783 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:49.389227 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:49.389232 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:49.391009 (+ 1777us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.126:45403: Server oryx/127.2.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":88,"threads_started":1}
W20260928 21:54:49.391171 11944 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:54:49.386608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.66:58845 (local address 127.2.182.126:45403)
0928 21:54:49.386653 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:49.386655 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:49.386749 (+ 94us) server_negotiation.cc:407] Connection header received
0928 21:54:49.386797 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:49.386798 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:49.386809 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:49.386858 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:49.387179 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:49.387735 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:49.388457 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:49.388670 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:49.388726 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:49.391111 (+ 2385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.66:58845: BlockingRecv error: recv got EOF from 127.2.182.66:58845 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 28 21:54:49 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.67@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:54:49 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.67@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:54:49.521356 11946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:49.510598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:45403 (local address 127.2.182.67:48339)
0928 21:54:49.510769 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:49.510779 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:49.510836 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:49.511077 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:49.511079 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:49.511240 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:49.511289 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:49.511292 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:49.512084 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:49.512085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:49.512814 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:49.512816 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:49.518604 (+ 5788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:49.518606 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:49.518687 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:49.519196 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:49.519202 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:49.521294 (+ 2092us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.126:45403: Server oryx/127.2.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":87,"threads_started":1}
W20260928 21:54:49.521469 11944 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:54:49.510716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.67:48339 (local address 127.2.182.126:45403)
0928 21:54:49.510762 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:49.510764 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:49.510824 (+ 60us) server_negotiation.cc:407] Connection header received
0928 21:54:49.510893 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:49.510893 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:49.510905 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:49.511019 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:49.511492 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:49.511998 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:49.512887 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:49.513258 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:49.513326 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:49.521404 (+ 8078us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.67:48339: BlockingRecv error: recv got EOF from 127.2.182.67:48339 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 28 21:54:50 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.65@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:54:50 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.65@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:54:50.275471 11947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:50.270798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:45403 (local address 127.2.182.65:46333)
0928 21:54:50.270952 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:50.270964 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:50.271028 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:50.271209 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:50.271210 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:50.271436 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:50.271488 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:50.271490 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:50.272159 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:50.272160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:50.272724 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:50.272726 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:50.273061 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:50.273062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:50.273111 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:50.273581 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:50.273586 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:50.275403 (+ 1817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.126:45403: Server oryx/127.2.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":58,"threads_started":1}
W20260928 21:54:50.275609 11948 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:54:50.270992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.65:46333 (local address 127.2.182.126:45403)
0928 21:54:50.271096 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:50.271098 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:50.271105 (+ 7us) server_negotiation.cc:407] Connection header received
0928 21:54:50.271126 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:50.271127 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:50.271136 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:50.271191 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:50.271545 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:50.272073 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:50.272792 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:50.273000 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:50.273050 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:50.275532 (+ 2482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.65:46333: BlockingRecv error: recv got EOF from 127.2.182.65:46333 (error 108)
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":47,"threads_started":1}
Sep 28 21:54:50 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.66@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:54:50 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.66@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:54:50.396461 11949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:50.391627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:45403 (local address 127.2.182.66:45823)
0928 21:54:50.391754 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:50.391765 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:50.391841 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:50.392060 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:50.392061 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:50.392216 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:50.392265 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:50.392267 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:50.393011 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:50.393012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:50.393572 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:50.393574 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:50.393999 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:50.394000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:50.394056 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:50.394514 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:50.394518 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:50.396389 (+ 1871us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.126:45403: Server oryx/127.2.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":58,"threads_started":1}
W20260928 21:54:50.396576 11948 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:54:50.391757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.66:45823 (local address 127.2.182.126:45403)
0928 21:54:50.391900 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:50.391902 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:50.391911 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:54:50.391938 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:50.391939 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:50.391949 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:50.392010 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:50.392365 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:50.392873 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:50.393659 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:50.393866 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:50.393979 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:50.396513 (+ 2534us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.66:45823: BlockingRecv error: recv got EOF from 127.2.182.66:45823 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 28 21:54:50 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.67@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:54:50 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.67@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:54:50.526106 11950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:50.521828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:45403 (local address 127.2.182.67:57551)
0928 21:54:50.521965 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:50.521975 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:50.522022 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:50.522131 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:50.522131 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:50.522281 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:50.522328 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:50.522329 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:50.522973 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:50.522974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:50.523559 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:50.523562 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:50.523911 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:50.523912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:50.523950 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:50.524344 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:50.524347 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:50.526027 (+ 1680us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.126:45403: Server oryx/127.2.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":82,"threads_started":1}
W20260928 21:54:50.526180 11948 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:54:50.521956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.67:57551 (local address 127.2.182.126:45403)
0928 21:54:50.522018 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:50.522020 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:50.522028 (+ 8us) server_negotiation.cc:407] Connection header received
0928 21:54:50.522064 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:50.522065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:50.522077 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:50.522119 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:50.522389 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:50.522904 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:50.523657 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:50.523847 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:50.523907 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:50.526127 (+ 2220us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.67:57551: BlockingRecv error: recv got EOF from 127.2.182.67:57551 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 28 21:54:51 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.65@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:54:51 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.65@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:54:51.281239 11951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:51.276119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:45403 (local address 127.2.182.65:32959)
0928 21:54:51.276266 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:51.276278 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:51.276329 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:51.276569 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:51.276571 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:51.276881 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:51.276939 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:51.276943 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:51.277662 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:51.277663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:51.278228 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:51.278230 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:51.278631 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:51.278633 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:51.278693 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:51.279188 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:51.279193 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:51.281170 (+ 1977us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.126:45403: Server oryx/127.2.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":65,"threads_started":1}
W20260928 21:54:51.281361 11952 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:54:51.276199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.65:32959 (local address 127.2.182.126:45403)
0928 21:54:51.276302 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:51.276304 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:51.276314 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:54:51.276414 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:51.276415 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:51.276429 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:51.276493 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:51.277040 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:51.277572 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:51.278320 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:51.278536 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:51.278587 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:51.281281 (+ 2694us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.65:32959: BlockingRecv error: recv got EOF from 127.2.182.65:32959 (error 108)
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":60,"threads_started":1}
Sep 28 21:54:51 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.66@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:54:51 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.66@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:54:51.401762 11953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:51.397071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:45403 (local address 127.2.182.66:51673)
0928 21:54:51.397192 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:51.397203 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:51.397259 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:51.397403 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:51.397403 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:51.397566 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:51.397611 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:51.397613 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:51.398383 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:51.398384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:51.399022 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:51.399025 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:51.399518 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:51.399519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:51.399581 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:51.400029 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:51.400034 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:51.401685 (+ 1651us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.126:45403: Server oryx/127.2.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":71,"threads_started":1}
W20260928 21:54:51.402143 11952 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:54:51.397188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.66:51673 (local address 127.2.182.126:45403)
0928 21:54:51.397282 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:51.397284 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:51.397294 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:54:51.397323 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:51.397324 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:51.397334 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:51.397396 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:51.397710 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:51.398274 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:51.399129 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:51.399397 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:51.399484 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:51.401971 (+ 2487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.66:51673: BlockingRecv error: recv got EOF from 127.2.182.66:51673 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 28 21:54:51 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.67@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:54:51 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.67@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:54:51.531390 11954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:51.526627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:45403 (local address 127.2.182.67:55763)
0928 21:54:51.526776 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:51.526788 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:51.526854 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:51.527038 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:51.527039 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:51.527199 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:51.527245 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:51.527246 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:51.528023 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:51.528024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:51.528590 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:51.528592 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:51.529047 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:51.529047 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:51.529092 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:51.529507 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:51.529511 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:51.531262 (+ 1751us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.126:45403: Server oryx/127.2.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":67,"threads_started":1}
W20260928 21:54:51.531451 11952 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:54:51.526757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.67:55763 (local address 127.2.182.126:45403)
0928 21:54:51.526830 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:51.526832 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:51.526843 (+ 11us) server_negotiation.cc:407] Connection header received
0928 21:54:51.526912 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:51.526913 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:51.526925 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:51.526963 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:51.527380 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:51.527927 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:51.528681 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:51.528899 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:51.529051 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:51.531386 (+ 2335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.67:55763: BlockingRecv error: recv got EOF from 127.2.182.67:55763 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 28 21:54:52 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.65@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:54:52 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.65@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:54:52.287019 11955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:52.281874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:45403 (local address 127.2.182.65:35639)
0928 21:54:52.282003 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:52.282014 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:52.282068 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:52.282192 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:52.282193 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:52.282408 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:52.282458 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:52.282460 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:52.283213 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:52.283214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:52.283929 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:52.283932 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:52.284325 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:52.284326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:52.284376 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:52.284827 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:52.284832 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:52.286944 (+ 2112us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.126:45403: Server oryx/127.2.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":63,"threads_started":1}
W20260928 21:54:52.287151 11956 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:54:52.281910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.65:35639 (local address 127.2.182.126:45403)
0928 21:54:52.282060 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:52.282062 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:52.282074 (+ 12us) server_negotiation.cc:407] Connection header received
0928 21:54:52.282106 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:52.282106 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:52.282118 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:52.282188 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:52.282556 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:52.283087 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:52.284025 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:52.284240 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:52.284302 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:52.287075 (+ 2773us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.65:35639: BlockingRecv error: recv got EOF from 127.2.182.65:35639 (error 108)
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":70,"threads_started":1}
Sep 28 21:54:52 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.66@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:54:52 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.66@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:54:52.407083 11957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:52.402258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:45403 (local address 127.2.182.66:52159)
0928 21:54:52.402413 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:52.402424 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:52.402489 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:52.402683 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:52.402684 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:52.402847 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:52.402891 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:52.402893 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:52.403671 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:52.403672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:52.404217 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:52.404219 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:52.404583 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:52.404584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:52.404633 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:52.405054 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:52.405058 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:52.407021 (+ 1963us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.126:45403: Server oryx/127.2.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":95,"threads_started":1}
W20260928 21:54:52.407212 11956 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:54:52.402361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.66:52159 (local address 127.2.182.126:45403)
0928 21:54:52.402417 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:52.402419 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:52.402479 (+ 60us) server_negotiation.cc:407] Connection header received
0928 21:54:52.402536 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:52.402537 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:52.402548 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:52.402607 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:52.403020 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:52.403596 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:52.404293 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:52.404501 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:52.404561 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:52.407140 (+ 2579us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.66:52159: BlockingRecv error: recv got EOF from 127.2.182.66:52159 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 28 21:54:52 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.67@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:54:52 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.67@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:54:52.536574 11958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:52.531856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:45403 (local address 127.2.182.67:46479)
0928 21:54:52.532031 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:52.532043 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:52.532108 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:52.532294 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:52.532296 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:52.532487 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:52.532532 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:52.532534 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:52.533237 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:52.533238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:52.533797 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:52.533799 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:52.534207 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:52.534208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:52.534255 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:52.534723 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:52.534728 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:52.536497 (+ 1769us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.126:45403: Server oryx/127.2.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":82,"threads_started":1}
W20260928 21:54:52.536649 11956 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:54:52.532027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.67:46479 (local address 127.2.182.126:45403)
0928 21:54:52.532124 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:52.532126 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:52.532135 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:54:52.532162 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:52.532163 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:52.532176 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:52.532232 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:52.532626 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:52.533157 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:52.533897 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:52.534096 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:52.534170 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:52.536594 (+ 2424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.67:46479: BlockingRecv error: recv got EOF from 127.2.182.67:46479 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 28 21:54:53 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.65@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:54:53 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.65@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:54:53.292706 11959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:53.287555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:45403 (local address 127.2.182.65:46853)
0928 21:54:53.287674 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:53.287685 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:53.287735 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:53.288102 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:53.288103 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:53.288256 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:53.288305 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:53.288307 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:53.289080 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:53.289082 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:53.289655 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:53.289657 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:53.290046 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:53.290047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:53.290098 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:53.290586 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:53.290591 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:53.292643 (+ 2052us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.126:45403: Server oryx/127.2.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":65,"threads_started":1}
W20260928 21:54:53.292845 11960 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:54:53.287668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.65:46853 (local address 127.2.182.126:45403)
0928 21:54:53.287960 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:53.287961 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:53.287969 (+ 8us) server_negotiation.cc:407] Connection header received
0928 21:54:53.287991 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:53.287992 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:53.288002 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:53.288065 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:53.288386 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:53.288976 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:53.289757 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:53.289967 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:53.290026 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:53.292765 (+ 2739us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.65:46853: BlockingRecv error: recv got EOF from 127.2.182.65:46853 (error 108)
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":78,"threads_started":1}
Sep 28 21:54:53 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.66@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:54:53 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.66@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:54:53.412878 11961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:53.407689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:45403 (local address 127.2.182.66:34451)
0928 21:54:53.407842 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:53.407854 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:53.407912 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:53.408169 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:53.408171 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:53.408357 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:53.408404 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:53.408406 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:53.409056 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:53.409057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:53.409663 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:53.409666 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:53.410120 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:53.410121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:53.410178 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:53.410709 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:53.410714 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:53.412787 (+ 2073us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.126:45403: Server oryx/127.2.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":85,"threads_started":1}
W20260928 21:54:53.412971 11960 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:54:53.407845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.66:34451 (local address 127.2.182.126:45403)
0928 21:54:53.408021 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:53.408024 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:53.408033 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:54:53.408058 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:53.408058 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:53.408070 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:53.408137 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:53.408485 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:53.408981 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:53.409755 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:53.410043 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:53.410104 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:53.412918 (+ 2814us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.66:34451: BlockingRecv error: recv got EOF from 127.2.182.66:34451 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
Sep 28 21:54:53 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.67@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:54:53 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.67@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:54:53.542112 11962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:53.537120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:45403 (local address 127.2.182.67:51509)
0928 21:54:53.537277 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:53.537291 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:53.537372 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:53.537541 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:53.537542 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:53.537708 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:53.537753 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:53.537754 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:53.538658 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:53.538659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:53.539217 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:53.539219 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:53.539618 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:53.539619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:53.539679 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:53.540215 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:53.540219 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:53.542031 (+ 1812us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.126:45403: Server oryx/127.2.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":80,"threads_started":1}
W20260928 21:54:53.542258 11960 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:54:53.537219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.67:51509 (local address 127.2.182.126:45403)
0928 21:54:53.537292 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:53.537294 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:53.537361 (+ 67us) server_negotiation.cc:407] Connection header received
0928 21:54:53.537462 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:53.537463 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:53.537478 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:53.537526 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:53.537829 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:53.538553 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:53.539292 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:53.539532 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:53.539598 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:53.542189 (+ 2591us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.67:51509: BlockingRecv error: recv got EOF from 127.2.182.67:51509 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 28 21:54:54 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.65@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:54:54 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.65@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:54:54.304316 11963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:54.293222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:45403 (local address 127.2.182.65:47215)
0928 21:54:54.293349 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:54.293359 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:54.293409 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:54.293601 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:54.293602 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:54.293781 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:54.293838 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:54.293840 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:54.294540 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:54.294541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:54.295112 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:54.295114 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:54.295525 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:54.295526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:54.295577 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:54.302305 (+ 6728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:54.302344 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:54.304247 (+ 1903us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.126:45403: Server oryx/127.2.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":60,"threads_started":1}
W20260928 21:54:54.304497 11964 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:54:54.293344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.65:47215 (local address 127.2.182.126:45403)
0928 21:54:54.293465 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:54.293467 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:54.293477 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:54:54.293502 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:54.293502 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:54.293513 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:54.293590 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:54.293900 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:54.294455 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:54.295209 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:54.295430 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:54.295493 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:54.304423 (+ 8930us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.65:47215: BlockingRecv error: recv got EOF from 127.2.182.65:47215 (error 108)
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":52,"threads_started":1}
Sep 28 21:54:54 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.66@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:54:54 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.66@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:54:54.418030 11965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:54.413463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:45403 (local address 127.2.182.66:50171)
0928 21:54:54.413578 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:54.413589 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:54.413647 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:54.413791 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:54.413791 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:54.413968 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:54.414015 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:54.414017 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:54.414689 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:54.414689 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:54.415216 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:54.415217 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:54.415632 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:54.415633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:54.415680 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:54.416150 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:54.416155 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:54.417964 (+ 1809us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.126:45403: Server oryx/127.2.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":66,"threads_started":1}
W20260928 21:54:54.418134 11964 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:54:54.413552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.66:50171 (local address 127.2.182.126:45403)
0928 21:54:54.413636 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:54.413638 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:54.413650 (+ 12us) server_negotiation.cc:407] Connection header received
0928 21:54:54.413693 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:54.413694 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:54.413704 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:54.413776 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:54.414088 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:54.414624 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:54.415294 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:54.415567 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:54.415621 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:54.418070 (+ 2449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.66:50171: BlockingRecv error: recv got EOF from 127.2.182.66:50171 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 28 21:54:54 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.67@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:54:54 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.67@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:54:54.547669 11966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:54.542622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:45403 (local address 127.2.182.67:59029)
0928 21:54:54.542751 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:54.542762 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:54.542808 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:54.543001 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:54.543002 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:54.543167 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:54.543212 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:54.543213 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:54.544071 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:54.544072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:54.544668 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:54.544670 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:54.545082 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:54.545083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:54.545133 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:54.545613 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:54.545618 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:54.547609 (+ 1991us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.126:45403: Server oryx/127.2.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":62,"threads_started":1}
W20260928 21:54:54.547751 11964 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:54:54.542762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.67:59029 (local address 127.2.182.126:45403)
0928 21:54:54.542825 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:54.542827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:54.542835 (+ 8us) server_negotiation.cc:407] Connection header received
0928 21:54:54.542857 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:54.542858 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:54.542870 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:54.542931 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:54.543313 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:54.543975 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:54.544763 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:54.544989 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:54.545052 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:54.547688 (+ 2636us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.67:59029: BlockingRecv error: recv got EOF from 127.2.182.67:59029 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 28 21:54:55 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.65@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:54:55 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.65@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:54:55.309576 11967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:55.304931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:45403 (local address 127.2.182.65:47853)
0928 21:54:55.305062 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:55.305073 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:55.305128 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:55.305259 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:55.305260 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:55.305421 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:55.305469 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:55.305470 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:55.306174 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:55.306175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:55.306737 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:55.306739 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:55.307124 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:55.307125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:55.307169 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:55.307658 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:55.307663 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:55.309515 (+ 1852us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.126:45403: Server oryx/127.2.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":75,"threads_started":1}
W20260928 21:54:55.309695 11968 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:54:55.305026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.65:47853 (local address 127.2.182.126:45403)
0928 21:54:55.305130 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:55.305131 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:55.305158 (+ 27us) server_negotiation.cc:407] Connection header received
0928 21:54:55.305180 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:55.305180 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:55.305190 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:55.305236 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:55.305524 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:55.306102 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:55.306860 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:55.307060 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:55.307111 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:55.309623 (+ 2512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.65:47853: BlockingRecv error: recv got EOF from 127.2.182.65:47853 (error 108)
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":45,"threads_started":1}
Sep 28 21:54:55 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.66@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:54:55 dist-test-slave-k6lz krb5kdc[10909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.182.66@KRBTEST.COM for oryx/127.2.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:54:55.423204 11969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:55.418571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:45403 (local address 127.2.182.66:40803)
0928 21:54:55.418682 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:55.418694 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:55.418759 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:55.418901 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:55.418902 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:55.419069 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:55.419116 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:55.419118 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:55.419916 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:55.419917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:55.420483 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:55.420484 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:55.420930 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:55.420931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:55.420982 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:55.421488 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:55.421492 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:55.423138 (+ 1646us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.182.126:45403: Server oryx/127.2.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":63,"threads_started":1}
W20260928 21:54:55.423293 11968 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:54:55.418604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.66:40803 (local address 127.2.182.126:45403)
0928 21:54:55.418658 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:55.418660 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:55.418762 (+ 102us) server_negotiation.cc:407] Connection header received
0928 21:54:55.418816 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:55.418817 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:55.418828 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:55.418863 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:55.419184 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:55.419717 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:55.420571 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:55.420768 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:55.420821 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:55.423230 (+ 2409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.182.66:40803: BlockingRecv error: recv got EOF from 127.2.182.66:40803 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 21:54:55.461635 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 11492
I20260928 21:54:55.465538 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 11645
I20260928 21:54:55.469260 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 11777
I20260928 21:54:55.472882 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 11422
2026-09-28T21:54:55Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17444 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-28T21:54:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:54:55Z Disabled control of system clock
I20260928 21:54:55.498652 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632440055392-2777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632440055392-2777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632440055392-2777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:42779
--webserver_interface=127.2.182.126
--webserver_port=0
--builtin_ntp_servers=127.2.182.84:35041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.126:42779
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 21:54:55.578323 11973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:54:55.578527 11973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:54:55.578547 11973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:54:55.579697 11973 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:54:55.579734 11973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:54:55.579746 11973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:54:55.579756 11973 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:54:55.579766 11973 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:54:55.581115 11973 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:35041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632440055392-2777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632440055392-2777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.126:42779
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:42779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632440055392-2777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:54:55.581333 11973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:54:55.581588 11973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:54:55.586649 11979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:55.586618 11981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:55.586665 11973 server_base.cc:1034] running on GCE node
W20260928 21:54:55.586633 11978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:55.587167 11973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:55.587487 11973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:55.588644 11973 hybrid_clock.cc:648] HybridClock initialized: now 1790632495588615 us; error 40 us; skew 500 ppm
I20260928 21:54:55.589982 11973 webserver.cc:460] Webserver started at http://127.2.182.126:39485/ using document root <none> and password file <none>
I20260928 21:54:55.590206 11973 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:55.590252 11973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:55.590370 11973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:54:55.591446 11973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632440055392-2777-0/minicluster-data/master-0/data/instance:
uuid: "05bb189fd60444509666d9963afac4a1"
format_stamp: "Formatted at 2026-09-28 21:54:55 on dist-test-slave-k6lz"
server_key: "859523e4ee163b4e3d3b7942623b451d"
server_key_iv: "c00017339413c3b76e1f5fd1eb413f34"
server_key_version: "encryptionkey@0"
I20260928 21:54:55.591737 11973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632440055392-2777-0/minicluster-data/master-0/wal/instance:
uuid: "05bb189fd60444509666d9963afac4a1"
format_stamp: "Formatted at 2026-09-28 21:54:55 on dist-test-slave-k6lz"
server_key: "859523e4ee163b4e3d3b7942623b451d"
server_key_iv: "c00017339413c3b76e1f5fd1eb413f34"
server_key_version: "encryptionkey@0"
I20260928 21:54:55.593077 11973 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:54:55.593832 11987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:55.594030 11973 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:54:55.594112 11973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632440055392-2777-0/minicluster-data/master-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632440055392-2777-0/minicluster-data/master-0/wal
uuid: "05bb189fd60444509666d9963afac4a1"
format_stamp: "Formatted at 2026-09-28 21:54:55 on dist-test-slave-k6lz"
server_key: "859523e4ee163b4e3d3b7942623b451d"
server_key_iv: "c00017339413c3b76e1f5fd1eb413f34"
server_key_version: "encryptionkey@0"
I20260928 21:54:55.594175 11973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632440055392-2777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632440055392-2777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632440055392-2777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:55.607527 11973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:54:55.607846 11973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:54:55.607987 11973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:54:55.611855 11973 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.126:42779
I20260928 21:54:55.611896 12039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.126:42779 every 8 connection(s)
I20260928 21:54:55.612229 11973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
I20260928 21:54:55.612869 12040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:54:55.613168 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 11973
I20260928 21:54:55.613257 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632440055392-2777-0/minicluster-data/master-0/wal/instance
I20260928 21:54:55.613442 2777 external_mini_cluster.cc:1428] Setting key afbf09cec43c11641711536848116f37
I20260928 21:54:55.616605 12040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05bb189fd60444509666d9963afac4a1: Bootstrap starting.
I20260928 21:54:55.617200 12040 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05bb189fd60444509666d9963afac4a1: Neither blocks nor log segments found. Creating new log.
I20260928 21:54:55.617448 12040 log.cc:824] T 00000000000000000000000000000000 P 05bb189fd60444509666d9963afac4a1: Log is configured to *not* fsync() on all Append() calls
I20260928 21:54:55.618065 12040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05bb189fd60444509666d9963afac4a1: No bootstrap required, opened a new log
I20260928 21:54:55.619136 12043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:55.614017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41048 (local address 127.2.182.126:42779)
0928 21:54:55.614236 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:55.614245 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:55.614280 (+ 35us) server_negotiation.cc:407] Connection header received
0928 21:54:55.614813 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:55.614823 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:55.614905 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:55.615053 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:55.615235 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:55.616574 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:55.617381 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:55.617818 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:55.617935 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:55.618565 (+ 630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:55.618569 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:55.618569 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:54:55.618731 (+ 162us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:55.618762 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:54:55.618772 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:55.618822 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:55.618895 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 21:54:55.618981 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":94,"threads_started":1}
I20260928 21:54:55.619997 12040 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05bb189fd60444509666d9963afac4a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05bb189fd60444509666d9963afac4a1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 42779 } }
I20260928 21:54:55.620139 12040 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05bb189fd60444509666d9963afac4a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:54:55.620173 12040 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05bb189fd60444509666d9963afac4a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05bb189fd60444509666d9963afac4a1, State: Initialized, Role: FOLLOWER
I20260928 21:54:55.620285 12040 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05bb189fd60444509666d9963afac4a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05bb189fd60444509666d9963afac4a1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 42779 } }
I20260928 21:54:55.620345 12040 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05bb189fd60444509666d9963afac4a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:54:55.620373 12040 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05bb189fd60444509666d9963afac4a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:54:55.620419 12040 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05bb189fd60444509666d9963afac4a1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:54:55.620949 12040 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05bb189fd60444509666d9963afac4a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05bb189fd60444509666d9963afac4a1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 42779 } }
I20260928 21:54:55.621044 12040 leader_election.cc:302] T 00000000000000000000000000000000 P 05bb189fd60444509666d9963afac4a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05bb189fd60444509666d9963afac4a1; no voters:
I20260928 21:54:55.621189 12040 leader_election.cc:288] T 00000000000000000000000000000000 P 05bb189fd60444509666d9963afac4a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:54:55.621236 12045 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05bb189fd60444509666d9963afac4a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:54:55.621373 12045 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05bb189fd60444509666d9963afac4a1 [term 1 LEADER]: Becoming Leader. State: Replica: 05bb189fd60444509666d9963afac4a1, State: Running, Role: LEADER
I20260928 21:54:55.621435 12040 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05bb189fd60444509666d9963afac4a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:54:55.621460 12045 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05bb189fd60444509666d9963afac4a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05bb189fd60444509666d9963afac4a1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 42779 } }
I20260928 21:54:55.621802 12046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05bb189fd60444509666d9963afac4a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "05bb189fd60444509666d9963afac4a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05bb189fd60444509666d9963afac4a1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 42779 } } }
I20260928 21:54:55.621819 12047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05bb189fd60444509666d9963afac4a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05bb189fd60444509666d9963afac4a1. Latest consensus state: current_term: 1 leader_uuid: "05bb189fd60444509666d9963afac4a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05bb189fd60444509666d9963afac4a1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 42779 } } }
I20260928 21:54:55.621944 12047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05bb189fd60444509666d9963afac4a1 [sys.catalog]: This master's current role is: LEADER
I20260928 21:54:55.622129 12053 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:54:55.622306 12046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05bb189fd60444509666d9963afac4a1 [sys.catalog]: This master's current role is: LEADER
I20260928 21:54:55.622617 12053 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:54:55.624153 12053 catalog_manager.cc:1347] Generated new cluster ID: b290ee13cc0d473c9dfd1f3dc52ad807
I20260928 21:54:55.624204 12053 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:54:55.641242 12053 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:54:55.641779 12053 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:54:55.644804 12053 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 05bb189fd60444509666d9963afac4a1: Generated new TSK 0
I20260928 21:54:55.645072 12053 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:54:55.653877 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.65:0
--local_ip_for_outbound_sockets=127.2.182.65
--webserver_interface=127.2.182.65
--webserver_port=0
--tserver_master_addrs=127.2.182.126:42779
--builtin_ntp_servers=127.2.182.84:35041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 21:54:55.738436 12064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:54:55.738646 12064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:54:55.738675 12064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:54:55.739878 12064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:54:55.739926 12064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:54:55.739961 12064 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.65
I20260928 21:54:55.741431 12064 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:35041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.182.65
--webserver_port=0
--tserver_master_addrs=127.2.182.126:42779
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.65
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:54:55.741657 12064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:54:55.741935 12064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:54:55.747362 12069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:55.747391 12072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:55.747471 12070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:55.747491 12064 server_base.cc:1034] running on GCE node
I20260928 21:54:55.747694 12064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:55.747895 12064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:55.749054 12064 hybrid_clock.cc:648] HybridClock initialized: now 1790632495749035 us; error 34 us; skew 500 ppm
I20260928 21:54:55.750432 12064 webserver.cc:460] Webserver started at http://127.2.182.65:43607/ using document root <none> and password file <none>
I20260928 21:54:55.750644 12064 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:55.750689 12064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:55.750814 12064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:54:55.751907 12064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-0/data/instance:
uuid: "7991f6b865894ed4b8072e713a1a7679"
format_stamp: "Formatted at 2026-09-28 21:54:55 on dist-test-slave-k6lz"
server_key: "28da4f2a6729f34531253874b41b05d5"
server_key_iv: "45f7cc8b09d8ed6236e319d12c991ae4"
server_key_version: "encryptionkey@0"
I20260928 21:54:55.752223 12064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-0/wal/instance:
uuid: "7991f6b865894ed4b8072e713a1a7679"
format_stamp: "Formatted at 2026-09-28 21:54:55 on dist-test-slave-k6lz"
server_key: "28da4f2a6729f34531253874b41b05d5"
server_key_iv: "45f7cc8b09d8ed6236e319d12c991ae4"
server_key_version: "encryptionkey@0"
I20260928 21:54:55.753453 12064 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:54:55.754279 12078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:55.754460 12064 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:54:55.754544 12064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-0/wal
uuid: "7991f6b865894ed4b8072e713a1a7679"
format_stamp: "Formatted at 2026-09-28 21:54:55 on dist-test-slave-k6lz"
server_key: "28da4f2a6729f34531253874b41b05d5"
server_key_iv: "45f7cc8b09d8ed6236e319d12c991ae4"
server_key_version: "encryptionkey@0"
I20260928 21:54:55.754606 12064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:55.762925 12064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:54:55.763213 12064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:54:55.763372 12064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:54:55.763607 12064 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:54:55.763918 12064 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:54:55.763952 12064 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:55.763980 12064 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:54:55.764006 12064 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:55.769531 12064 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.65:39361
I20260928 21:54:55.769594 12191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.65:39361 every 8 connection(s)
I20260928 21:54:55.769889 12064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
I20260928 21:54:55.772393 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 12064
I20260928 21:54:55.772480 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-0/wal/instance
I20260928 21:54:55.772686 2777 external_mini_cluster.cc:1428] Setting key 02f065004d03d96f1b0f125e9e312fff
I20260928 21:54:55.773581 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.66:0
--local_ip_for_outbound_sockets=127.2.182.66
--webserver_interface=127.2.182.66
--webserver_port=0
--tserver_master_addrs=127.2.182.126:42779
--builtin_ntp_servers=127.2.182.84:35041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 21:54:55.775674 12043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:55.770638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.65:36111 (local address 127.2.182.126:42779)
0928 21:54:55.770713 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:55.770715 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:55.770943 (+ 228us) server_negotiation.cc:407] Connection header received
0928 21:54:55.771523 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:55.771525 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:55.771540 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:55.771596 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:55.772240 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:55.772963 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:55.774191 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:55.774409 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:55.774465 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:55.775412 (+ 947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:55.775413 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:55.775413 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:54:55.775418 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:55.775432 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:54:55.775436 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:55.775479 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:55.775563 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 21:54:55.775594 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 21:54:55.775880 12194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:55.770608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:42779 (local address 127.2.182.65:36111)
0928 21:54:55.770804 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:55.770849 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:55.771431 (+ 582us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:55.771647 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:55.771651 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:55.771659 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:55.772124 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:55.772129 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:55.773047 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:55.773049 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:55.774110 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:55.774113 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:55.774479 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:55.774480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:55.774618 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:55.775197 (+ 579us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:54:55.775198 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:55.775227 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:54:55.775252 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:55.775490 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:55.775496 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:55.775499 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:54:55.775562 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 21:54:55.775665 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":93,"threads_started":1}
I20260928 21:54:55.776317 12192 heartbeater.cc:347] Connected to a master server at 127.2.182.126:42779
I20260928 21:54:55.776407 12192 heartbeater.cc:464] Registering TS with master...
I20260928 21:54:55.776582 12192 heartbeater.cc:511] Master 127.2.182.126:42779 requested a full tablet report, sending...
I20260928 21:54:55.777168 12004 ts_manager.cc:194] Registered new tserver with Master: 7991f6b865894ed4b8072e713a1a7679 (127.2.182.65:39361)
I20260928 21:54:55.778075 12004 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.182.65:36111
W20260928 21:54:55.854382 12195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:54:55.854604 12195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:54:55.854626 12195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:54:55.855808 12195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:54:55.855850 12195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:54:55.855882 12195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.66
I20260928 21:54:55.857251 12195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:35041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.182.66
--webserver_port=0
--tserver_master_addrs=127.2.182.126:42779
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.66
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:54:55.857467 12195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:54:55.857755 12195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:54:55.862650 12203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:55.862715 12200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:55.862649 12201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:55.862679 12195 server_base.cc:1034] running on GCE node
I20260928 21:54:55.862957 12195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:55.863176 12195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:55.864328 12195 hybrid_clock.cc:648] HybridClock initialized: now 1790632495864307 us; error 28 us; skew 500 ppm
I20260928 21:54:55.865670 12195 webserver.cc:460] Webserver started at http://127.2.182.66:38541/ using document root <none> and password file <none>
I20260928 21:54:55.865877 12195 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:55.865917 12195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:55.866007 12195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:54:55.867007 12195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-1/data/instance:
uuid: "68283782b10740c5ad00e1c57192f02d"
format_stamp: "Formatted at 2026-09-28 21:54:55 on dist-test-slave-k6lz"
server_key: "bf0aeb050e149deb994d5efd8fb8b27a"
server_key_iv: "3ae5cd82b32c28a9eeb3756ead1e9939"
server_key_version: "encryptionkey@0"
I20260928 21:54:55.867291 12195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-1/wal/instance:
uuid: "68283782b10740c5ad00e1c57192f02d"
format_stamp: "Formatted at 2026-09-28 21:54:55 on dist-test-slave-k6lz"
server_key: "bf0aeb050e149deb994d5efd8fb8b27a"
server_key_iv: "3ae5cd82b32c28a9eeb3756ead1e9939"
server_key_version: "encryptionkey@0"
I20260928 21:54:55.868652 12195 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:54:55.869459 12209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:55.869669 12195 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:54:55.869751 12195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-1/wal
uuid: "68283782b10740c5ad00e1c57192f02d"
format_stamp: "Formatted at 2026-09-28 21:54:55 on dist-test-slave-k6lz"
server_key: "bf0aeb050e149deb994d5efd8fb8b27a"
server_key_iv: "3ae5cd82b32c28a9eeb3756ead1e9939"
server_key_version: "encryptionkey@0"
I20260928 21:54:55.869808 12195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:55.889931 12195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:54:55.890249 12195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:54:55.890386 12195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:54:55.890594 12195 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:54:55.890929 12195 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:54:55.890964 12195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:55.890992 12195 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:54:55.891009 12195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:55.896528 12195 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.66:43693
I20260928 21:54:55.896589 12322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.66:43693 every 8 connection(s)
I20260928 21:54:55.896937 12195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
I20260928 21:54:55.898567 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 12195
I20260928 21:54:55.898660 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-1/wal/instance
I20260928 21:54:55.898859 2777 external_mini_cluster.cc:1428] Setting key 9520c12f243eb7c1b36774d7a5929850
I20260928 21:54:55.899907 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.67:0
--local_ip_for_outbound_sockets=127.2.182.67
--webserver_interface=127.2.182.67
--webserver_port=0
--tserver_master_addrs=127.2.182.126:42779
--builtin_ntp_servers=127.2.182.84:35041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 21:54:55.902454 12043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:55.897596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.66:54147 (local address 127.2.182.126:42779)
0928 21:54:55.897645 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:55.897647 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:55.897881 (+ 234us) server_negotiation.cc:407] Connection header received
0928 21:54:55.898410 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:55.898411 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:55.898423 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:55.898482 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:55.899302 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:55.899902 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:55.900911 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:55.901208 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:55.901327 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:55.902195 (+ 868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:55.902197 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:55.902197 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:54:55.902202 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:55.902225 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:54:55.902229 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:55.902285 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:55.902347 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 21:54:55.902377 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 21:54:55.902613 12325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:55.897582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:42779 (local address 127.2.182.66:54147)
0928 21:54:55.897767 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:55.897807 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:55.898281 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:55.898571 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:55.898576 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:55.898588 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:55.899221 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:55.899226 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:55.900003 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:55.900004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:55.900823 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:55.900825 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:55.901340 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:55.901341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:55.901513 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:55.902051 (+ 538us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:54:55.902051 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:55.902079 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:54:55.902102 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:55.902289 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:55.902291 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:55.902293 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 21:54:55.902337 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 21:54:55.902419 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":69,"threads_started":1}
I20260928 21:54:55.903141 12323 heartbeater.cc:347] Connected to a master server at 127.2.182.126:42779
I20260928 21:54:55.903223 12323 heartbeater.cc:464] Registering TS with master...
I20260928 21:54:55.903427 12323 heartbeater.cc:511] Master 127.2.182.126:42779 requested a full tablet report, sending...
I20260928 21:54:55.903772 12004 ts_manager.cc:194] Registered new tserver with Master: 68283782b10740c5ad00e1c57192f02d (127.2.182.66:43693)
I20260928 21:54:55.904114 12004 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.182.66:54147
W20260928 21:54:55.980476 12326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:54:55.980700 12326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:54:55.980729 12326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:54:55.981959 12326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:54:55.982007 12326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:54:55.982041 12326 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.67
I20260928 21:54:55.983513 12326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:35041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.182.67
--webserver_port=0
--tserver_master_addrs=127.2.182.126:42779
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12326
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.67
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:54:55.983765 12326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:54:55.984052 12326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:54:55.989223 12332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:55.989221 12334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:55.989208 12331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:55.989239 12326 server_base.cc:1034] running on GCE node
I20260928 21:54:55.989661 12326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:55.989885 12326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:55.991025 12326 hybrid_clock.cc:648] HybridClock initialized: now 1790632495991012 us; error 28 us; skew 500 ppm
I20260928 21:54:55.992323 12326 webserver.cc:460] Webserver started at http://127.2.182.67:34551/ using document root <none> and password file <none>
I20260928 21:54:55.992571 12326 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:55.992620 12326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:55.992744 12326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:54:55.993791 12326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-2/data/instance:
uuid: "ad52e9d037694c79b1da816a7ecb57f4"
format_stamp: "Formatted at 2026-09-28 21:54:55 on dist-test-slave-k6lz"
server_key: "fc69522691b3c500cf17b681c19a67a6"
server_key_iv: "6979efdef022b405ac38c6c2888f7b2c"
server_key_version: "encryptionkey@0"
I20260928 21:54:55.994231 12326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-2/wal/instance:
uuid: "ad52e9d037694c79b1da816a7ecb57f4"
format_stamp: "Formatted at 2026-09-28 21:54:55 on dist-test-slave-k6lz"
server_key: "fc69522691b3c500cf17b681c19a67a6"
server_key_iv: "6979efdef022b405ac38c6c2888f7b2c"
server_key_version: "encryptionkey@0"
I20260928 21:54:55.995540 12326 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 21:54:55.996472 12340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:55.996690 12326 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:54:55.996780 12326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-2/wal
uuid: "ad52e9d037694c79b1da816a7ecb57f4"
format_stamp: "Formatted at 2026-09-28 21:54:55 on dist-test-slave-k6lz"
server_key: "fc69522691b3c500cf17b681c19a67a6"
server_key_iv: "6979efdef022b405ac38c6c2888f7b2c"
server_key_version: "encryptionkey@0"
I20260928 21:54:55.996862 12326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:56.005694 12326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:54:56.006014 12326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:54:56.006138 12326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:54:56.006357 12326 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:54:56.006678 12326 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:54:56.006711 12326 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:56.006739 12326 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:54:56.006764 12326 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:56.012275 12326 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.67:45633
I20260928 21:54:56.012326 12453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.67:45633 every 8 connection(s)
I20260928 21:54:56.012639 12326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
I20260928 21:54:56.014984 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 12326
I20260928 21:54:56.015086 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-2/wal/instance
I20260928 21:54:56.015398 2777 external_mini_cluster.cc:1428] Setting key d643780cbb99ef2ae53d9cabebb04d8c
I20260928 21:54:56.019294 12043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:56.013441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.67:48827 (local address 127.2.182.126:42779)
0928 21:54:56.013512 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:56.013514 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:56.013701 (+ 187us) server_negotiation.cc:407] Connection header received
0928 21:54:56.014179 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:56.014180 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:56.014191 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:56.014254 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:56.015393 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:56.016171 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:56.017618 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:56.017937 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:56.018024 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:56.018999 (+ 975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:56.019001 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:56.019002 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:54:56.019007 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:56.019023 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:54:56.019027 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:56.019054 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:56.019180 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 21:54:56.019209 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 21:54:56.019932 12456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:56.013383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:42779 (local address 127.2.182.67:48827)
0928 21:54:56.013570 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:56.013609 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:56.014099 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:56.014550 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:56.014555 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:56.014565 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:56.015237 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:56.015244 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:56.016275 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:56.016276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:56.017515 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:56.017518 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:56.018027 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:56.018028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:56.018180 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:56.018828 (+ 648us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:54:56.018828 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:56.018868 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:54:56.018899 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:56.019094 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:56.019099 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:56.019102 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:54:56.019599 (+ 497us) client_negotiation.cc:240] Negotiation successful
0928 21:54:56.019714 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":87,"threads_started":1}
I20260928 21:54:56.020411 12454 heartbeater.cc:347] Connected to a master server at 127.2.182.126:42779
I20260928 21:54:56.020529 12454 heartbeater.cc:464] Registering TS with master...
I20260928 21:54:56.020773 12454 heartbeater.cc:511] Master 127.2.182.126:42779 requested a full tablet report, sending...
I20260928 21:54:56.021220 12004 ts_manager.cc:194] Registered new tserver with Master: ad52e9d037694c79b1da816a7ecb57f4 (127.2.182.67:45633)
I20260928 21:54:56.021637 12004 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.182.67:48827
I20260928 21:54:56.029882 2777 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 21:54:56.031714 12043 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:54:56.031185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41056 (local address 127.2.182.126:42779)
0928 21:54:56.031227 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:56.031229 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:56.031308 (+ 79us) server_negotiation.cc:407] Connection header received
0928 21:54:56.031411 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:56.031411 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:56.031422 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:56.031477 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:56.031643 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41056: BlockingRecv error: recv got EOF from 127.0.0.1:41056 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 21:54:56.031785 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 12064
I20260928 21:54:56.036145 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 12195
I20260928 21:54:56.039805 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 12326
I20260928 21:54:56.043519 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 11973
2026-09-28T21:54:56Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (569 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-28T21:54:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:54:56Z Disabled control of system clock
I20260928 21:54:56.068233 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632440055392-2777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632440055392-2777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632440055392-2777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:44425
--webserver_interface=127.2.182.126
--webserver_port=0
--builtin_ntp_servers=127.2.182.84:37507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.126:44425
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 21:54:56.148833 12465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:54:56.149037 12465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:54:56.149058 12465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:54:56.150211 12465 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:54:56.150252 12465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:54:56.150264 12465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:54:56.150275 12465 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:54:56.150286 12465 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:54:56.151656 12465 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:37507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632440055392-2777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632440055392-2777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.126:44425
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:44425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632440055392-2777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:54:56.151873 12465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:54:56.152169 12465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:54:56.156980 12470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:56.156980 12473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:56.157033 12471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:56.157644 12465 server_base.cc:1034] running on GCE node
I20260928 21:54:56.157898 12465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:56.158259 12465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:56.159394 12465 hybrid_clock.cc:648] HybridClock initialized: now 1790632496159382 us; error 29 us; skew 500 ppm
I20260928 21:54:56.160706 12465 webserver.cc:460] Webserver started at http://127.2.182.126:37003/ using document root <none> and password file <none>
I20260928 21:54:56.160965 12465 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:56.161096 12465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:56.161224 12465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:54:56.162362 12465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632440055392-2777-0/minicluster-data/master-0/data/instance:
uuid: "84a6042ce3794657b6c8c4bb11f7bc3d"
format_stamp: "Formatted at 2026-09-28 21:54:56 on dist-test-slave-k6lz"
server_key: "8d6bfb4eda2417694811375337a1d4f2"
server_key_iv: "0102add274f7bb5f79c6931f55469d6c"
server_key_version: "encryptionkey@0"
I20260928 21:54:56.162696 12465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632440055392-2777-0/minicluster-data/master-0/wal/instance:
uuid: "84a6042ce3794657b6c8c4bb11f7bc3d"
format_stamp: "Formatted at 2026-09-28 21:54:56 on dist-test-slave-k6lz"
server_key: "8d6bfb4eda2417694811375337a1d4f2"
server_key_iv: "0102add274f7bb5f79c6931f55469d6c"
server_key_version: "encryptionkey@0"
I20260928 21:54:56.164088 12465 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:54:56.164928 12479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:56.165167 12465 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:54:56.165249 12465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632440055392-2777-0/minicluster-data/master-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632440055392-2777-0/minicluster-data/master-0/wal
uuid: "84a6042ce3794657b6c8c4bb11f7bc3d"
format_stamp: "Formatted at 2026-09-28 21:54:56 on dist-test-slave-k6lz"
server_key: "8d6bfb4eda2417694811375337a1d4f2"
server_key_iv: "0102add274f7bb5f79c6931f55469d6c"
server_key_version: "encryptionkey@0"
I20260928 21:54:56.165324 12465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632440055392-2777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632440055392-2777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632440055392-2777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:56.168751 12465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:54:56.169005 12465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:54:56.169118 12465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:54:56.172894 12465 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.126:44425
I20260928 21:54:56.172920 12531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.126:44425 every 8 connection(s)
I20260928 21:54:56.173228 12465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
I20260928 21:54:56.173831 12532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:54:56.176489 12532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84a6042ce3794657b6c8c4bb11f7bc3d: Bootstrap starting.
I20260928 21:54:56.177006 12532 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84a6042ce3794657b6c8c4bb11f7bc3d: Neither blocks nor log segments found. Creating new log.
I20260928 21:54:56.177256 12532 log.cc:824] T 00000000000000000000000000000000 P 84a6042ce3794657b6c8c4bb11f7bc3d: Log is configured to *not* fsync() on all Append() calls
I20260928 21:54:56.177845 12532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84a6042ce3794657b6c8c4bb11f7bc3d: No bootstrap required, opened a new log
I20260928 21:54:56.179467 12532 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84a6042ce3794657b6c8c4bb11f7bc3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84a6042ce3794657b6c8c4bb11f7bc3d" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 44425 } }
I20260928 21:54:56.179600 12532 raft_consensus.cc:374] T 00000000000000000000000000000000 P 84a6042ce3794657b6c8c4bb11f7bc3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:54:56.179620 12532 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84a6042ce3794657b6c8c4bb11f7bc3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84a6042ce3794657b6c8c4bb11f7bc3d, State: Initialized, Role: FOLLOWER
I20260928 21:54:56.179740 12532 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84a6042ce3794657b6c8c4bb11f7bc3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84a6042ce3794657b6c8c4bb11f7bc3d" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 44425 } }
I20260928 21:54:56.179808 12532 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84a6042ce3794657b6c8c4bb11f7bc3d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:54:56.179831 12532 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84a6042ce3794657b6c8c4bb11f7bc3d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:54:56.179876 12532 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84a6042ce3794657b6c8c4bb11f7bc3d [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:54:56.180390 12532 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84a6042ce3794657b6c8c4bb11f7bc3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84a6042ce3794657b6c8c4bb11f7bc3d" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 44425 } }
I20260928 21:54:56.180472 12532 leader_election.cc:302] T 00000000000000000000000000000000 P 84a6042ce3794657b6c8c4bb11f7bc3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84a6042ce3794657b6c8c4bb11f7bc3d; no voters:
I20260928 21:54:56.180610 12532 leader_election.cc:288] T 00000000000000000000000000000000 P 84a6042ce3794657b6c8c4bb11f7bc3d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:54:56.180670 12535 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84a6042ce3794657b6c8c4bb11f7bc3d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:54:56.180830 12535 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84a6042ce3794657b6c8c4bb11f7bc3d [term 1 LEADER]: Becoming Leader. State: Replica: 84a6042ce3794657b6c8c4bb11f7bc3d, State: Running, Role: LEADER
I20260928 21:54:56.180850 12532 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84a6042ce3794657b6c8c4bb11f7bc3d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:54:56.180924 12535 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84a6042ce3794657b6c8c4bb11f7bc3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84a6042ce3794657b6c8c4bb11f7bc3d" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 44425 } }
I20260928 21:54:56.181282 12537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84a6042ce3794657b6c8c4bb11f7bc3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84a6042ce3794657b6c8c4bb11f7bc3d. Latest consensus state: current_term: 1 leader_uuid: "84a6042ce3794657b6c8c4bb11f7bc3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84a6042ce3794657b6c8c4bb11f7bc3d" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 44425 } } }
I20260928 21:54:56.181360 12537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84a6042ce3794657b6c8c4bb11f7bc3d [sys.catalog]: This master's current role is: LEADER
I20260928 21:54:56.181566 12544 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:54:56.181609 12536 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84a6042ce3794657b6c8c4bb11f7bc3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "84a6042ce3794657b6c8c4bb11f7bc3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84a6042ce3794657b6c8c4bb11f7bc3d" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 44425 } } }
I20260928 21:54:56.181926 12536 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84a6042ce3794657b6c8c4bb11f7bc3d [sys.catalog]: This master's current role is: LEADER
I20260928 21:54:56.182245 12544 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:54:56.182507 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 12465
I20260928 21:54:56.182601 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632440055392-2777-0/minicluster-data/master-0/wal/instance
I20260928 21:54:56.182760 2777 external_mini_cluster.cc:1428] Setting key a741d164f00e3d43623b1d791d8bfed8
I20260928 21:54:56.184476 12544 catalog_manager.cc:1347] Generated new cluster ID: e4221e55b4fc49f6a9356eb9a79d760b
I20260928 21:54:56.184526 12544 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:54:56.185195 12554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:56.183184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51646 (local address 127.2.182.126:44425)
0928 21:54:56.183425 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:56.183434 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:56.183466 (+ 32us) server_negotiation.cc:407] Connection header received
0928 21:54:56.183576 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:56.183584 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:56.183637 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:56.183761 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:56.184770 (+ 1009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:56.184773 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:56.184774 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:54:56.184778 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:56.184826 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:54:56.184840 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:56.184908 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:56.184962 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 21:54:56.185019 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":102,"threads_started":1}
I20260928 21:54:56.202260 12544 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:54:56.202435 12544 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:54:56.211695 12544 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 84a6042ce3794657b6c8c4bb11f7bc3d: Generated new TSK 0
I20260928 21:54:56.211894 12544 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:54:56.215672 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.65:0
--local_ip_for_outbound_sockets=127.2.182.65
--webserver_interface=127.2.182.65
--webserver_port=0
--tserver_master_addrs=127.2.182.126:44425
--builtin_ntp_servers=127.2.182.84:37507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 21:54:56.294234 12556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:54:56.294452 12556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:54:56.294476 12556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:54:56.295689 12556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:54:56.295732 12556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:54:56.295758 12556 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.65
I20260928 21:54:56.297127 12556 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:37507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.182.65
--webserver_port=0
--tserver_master_addrs=127.2.182.126:44425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12556
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.65
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:54:56.297359 12556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:54:56.297653 12556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:54:56.302392 12561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:56.302497 12562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:56.302594 12564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:56.302739 12556 server_base.cc:1034] running on GCE node
I20260928 21:54:56.303017 12556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:56.303246 12556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:56.304383 12556 hybrid_clock.cc:648] HybridClock initialized: now 1790632496304371 us; error 31 us; skew 500 ppm
I20260928 21:54:56.305539 12556 webserver.cc:460] Webserver started at http://127.2.182.65:39185/ using document root <none> and password file <none>
I20260928 21:54:56.305757 12556 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:56.305804 12556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:56.305999 12556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:54:56.307049 12556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-0/data/instance:
uuid: "e3c240058e7b49bfa430f5b920e7ee8a"
format_stamp: "Formatted at 2026-09-28 21:54:56 on dist-test-slave-k6lz"
server_key: "2ca05d417c42b4ee0010f973af19dd80"
server_key_iv: "4b89009534c41d880216260cbb8e2f77"
server_key_version: "encryptionkey@0"
I20260928 21:54:56.307418 12556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-0/wal/instance:
uuid: "e3c240058e7b49bfa430f5b920e7ee8a"
format_stamp: "Formatted at 2026-09-28 21:54:56 on dist-test-slave-k6lz"
server_key: "2ca05d417c42b4ee0010f973af19dd80"
server_key_iv: "4b89009534c41d880216260cbb8e2f77"
server_key_version: "encryptionkey@0"
I20260928 21:54:56.308661 12556 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:54:56.309563 12570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:56.309734 12556 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:54:56.309836 12556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-0/wal
uuid: "e3c240058e7b49bfa430f5b920e7ee8a"
format_stamp: "Formatted at 2026-09-28 21:54:56 on dist-test-slave-k6lz"
server_key: "2ca05d417c42b4ee0010f973af19dd80"
server_key_iv: "4b89009534c41d880216260cbb8e2f77"
server_key_version: "encryptionkey@0"
I20260928 21:54:56.310000 12556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:56.313326 12556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:54:56.313563 12556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:54:56.313659 12556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:54:56.313833 12556 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:54:56.314092 12556 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:54:56.314119 12556 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:56.314145 12556 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:54:56.314162 12556 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:56.319489 12556 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.65:44305
I20260928 21:54:56.319538 12683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.65:44305 every 8 connection(s)
I20260928 21:54:56.319849 12556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
I20260928 21:54:56.320058 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 12556
I20260928 21:54:56.320152 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-0/wal/instance
I20260928 21:54:56.320341 2777 external_mini_cluster.cc:1428] Setting key 068a776b56689ec42a3ad3598533f7aa
I20260928 21:54:56.321381 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.66:0
--local_ip_for_outbound_sockets=127.2.182.66
--webserver_interface=127.2.182.66
--webserver_port=0
--tserver_master_addrs=127.2.182.126:44425
--builtin_ntp_servers=127.2.182.84:37507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 21:54:56.323225 12554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:56.321038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.65:38353 (local address 127.2.182.126:44425)
0928 21:54:56.321084 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:56.321086 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:56.321094 (+ 8us) server_negotiation.cc:407] Connection header received
0928 21:54:56.321681 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:56.321682 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:56.321689 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:56.321733 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:56.322996 (+ 1263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:56.322997 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:56.322997 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:54:56.323002 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:56.323018 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:54:56.323022 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:56.323071 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:56.323141 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 21:54:56.323156 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 21:54:56.323457 12686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:56.320706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:44425 (local address 127.2.182.65:38353)
0928 21:54:56.320901 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:56.320947 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:56.321590 (+ 643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:56.321805 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:56.321809 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:56.321819 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:56.322850 (+ 1031us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:54:56.322850 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:56.322880 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:54:56.322909 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:56.323076 (+ 167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:56.323081 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:56.323084 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:54:56.323151 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 21:54:56.323210 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":87,"threads_started":1}
I20260928 21:54:56.324069 12684 heartbeater.cc:347] Connected to a master server at 127.2.182.126:44425
I20260928 21:54:56.324157 12684 heartbeater.cc:464] Registering TS with master...
I20260928 21:54:56.324308 12684 heartbeater.cc:511] Master 127.2.182.126:44425 requested a full tablet report, sending...
I20260928 21:54:56.324842 12496 ts_manager.cc:194] Registered new tserver with Master: e3c240058e7b49bfa430f5b920e7ee8a (127.2.182.65:44305)
W20260928 21:54:56.403471 12687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:54:56.403689 12687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:54:56.403713 12687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:54:56.404971 12687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:54:56.405022 12687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:54:56.405061 12687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.66
I20260928 21:54:56.406560 12687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:37507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.182.66
--webserver_port=0
--tserver_master_addrs=127.2.182.126:44425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.66
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:54:56.406836 12687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:54:56.407156 12687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:54:56.412160 12692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:56.412357 12687 server_base.cc:1034] running on GCE node
W20260928 21:54:56.412124 12695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:56.412242 12693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:56.412698 12687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:56.412959 12687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:56.414129 12687 hybrid_clock.cc:648] HybridClock initialized: now 1790632496414120 us; error 31 us; skew 500 ppm
I20260928 21:54:56.415437 12687 webserver.cc:460] Webserver started at http://127.2.182.66:40871/ using document root <none> and password file <none>
I20260928 21:54:56.415648 12687 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:56.415696 12687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:56.415817 12687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:54:56.416955 12687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-1/data/instance:
uuid: "d25ce64bef8347fa88e9328127173e24"
format_stamp: "Formatted at 2026-09-28 21:54:56 on dist-test-slave-k6lz"
server_key: "a643b649affac47053a0d102b02df6b8"
server_key_iv: "b51e355a0269d0e4ac29e1e732f85b10"
server_key_version: "encryptionkey@0"
I20260928 21:54:56.417292 12687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-1/wal/instance:
uuid: "d25ce64bef8347fa88e9328127173e24"
format_stamp: "Formatted at 2026-09-28 21:54:56 on dist-test-slave-k6lz"
server_key: "a643b649affac47053a0d102b02df6b8"
server_key_iv: "b51e355a0269d0e4ac29e1e732f85b10"
server_key_version: "encryptionkey@0"
I20260928 21:54:56.418727 12687 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 21:54:56.419657 12701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:56.419821 12687 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:54:56.419975 12687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-1/wal
uuid: "d25ce64bef8347fa88e9328127173e24"
format_stamp: "Formatted at 2026-09-28 21:54:56 on dist-test-slave-k6lz"
server_key: "a643b649affac47053a0d102b02df6b8"
server_key_iv: "b51e355a0269d0e4ac29e1e732f85b10"
server_key_version: "encryptionkey@0"
I20260928 21:54:56.420042 12687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:56.423269 12687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:54:56.423547 12687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:54:56.423659 12687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:54:56.423848 12687 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:54:56.424144 12687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:54:56.424173 12687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:56.424196 12687 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:54:56.424217 12687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:56.429682 12687 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.66:33111
I20260928 21:54:56.429854 12814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.66:33111 every 8 connection(s)
I20260928 21:54:56.430081 12687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
I20260928 21:54:56.433362 12554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:56.430918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.66:40539 (local address 127.2.182.126:44425)
0928 21:54:56.431075 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:56.431077 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:56.431242 (+ 165us) server_negotiation.cc:407] Connection header received
0928 21:54:56.431775 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:56.431776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:56.431783 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:56.431831 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:56.433070 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:56.433072 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:56.433072 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:54:56.433078 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:56.433098 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:54:56.433103 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:56.433161 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:56.433233 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 21:54:56.433248 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260928 21:54:56.433530 12817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:56.430787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:44425 (local address 127.2.182.66:40539)
0928 21:54:56.431113 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:56.431157 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:56.431679 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:56.431912 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:56.431916 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:56.431926 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:56.432895 (+ 969us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:54:56.432896 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:56.432927 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:54:56.432957 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:56.433165 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:56.433171 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:56.433174 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:54:56.433245 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 21:54:56.433312 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":203,"threads_started":1}
I20260928 21:54:56.434124 12815 heartbeater.cc:347] Connected to a master server at 127.2.182.126:44425
I20260928 21:54:56.434219 12815 heartbeater.cc:464] Registering TS with master...
I20260928 21:54:56.434345 12815 heartbeater.cc:511] Master 127.2.182.126:44425 requested a full tablet report, sending...
I20260928 21:54:56.434702 12496 ts_manager.cc:194] Registered new tserver with Master: d25ce64bef8347fa88e9328127173e24 (127.2.182.66:33111)
I20260928 21:54:56.436654 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 12687
I20260928 21:54:56.436735 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-1/wal/instance
I20260928 21:54:56.437000 2777 external_mini_cluster.cc:1428] Setting key 8c699c6385d0ee5a798afb289a07dc92
I20260928 21:54:56.438002 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.67:0
--local_ip_for_outbound_sockets=127.2.182.67
--webserver_interface=127.2.182.67
--webserver_port=0
--tserver_master_addrs=127.2.182.126:44425
--builtin_ntp_servers=127.2.182.84:37507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 21:54:56.517966 12818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:54:56.518189 12818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:54:56.518219 12818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:54:56.519428 12818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:54:56.519469 12818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:54:56.519502 12818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.67
I20260928 21:54:56.520876 12818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:37507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.182.67
--webserver_port=0
--tserver_master_addrs=127.2.182.126:44425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.67
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:54:56.521127 12818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:54:56.521422 12818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:54:56.526507 12823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:56.526612 12824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:56.526599 12826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:56.526736 12818 server_base.cc:1034] running on GCE node
I20260928 21:54:56.526993 12818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:56.527196 12818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:56.528323 12818 hybrid_clock.cc:648] HybridClock initialized: now 1790632496528304 us; error 36 us; skew 500 ppm
I20260928 21:54:56.529572 12818 webserver.cc:460] Webserver started at http://127.2.182.67:40161/ using document root <none> and password file <none>
I20260928 21:54:56.529800 12818 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:56.529844 12818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:56.530020 12818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:54:56.531097 12818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-2/data/instance:
uuid: "12c9b4a5a5524132bfd9a35f43b634af"
format_stamp: "Formatted at 2026-09-28 21:54:56 on dist-test-slave-k6lz"
server_key: "9b645e0b72fd20d5dbb05b94ab9ee6bb"
server_key_iv: "0b4f3476e308447eb0d70d16600a54c1"
server_key_version: "encryptionkey@0"
I20260928 21:54:56.531455 12818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-2/wal/instance:
uuid: "12c9b4a5a5524132bfd9a35f43b634af"
format_stamp: "Formatted at 2026-09-28 21:54:56 on dist-test-slave-k6lz"
server_key: "9b645e0b72fd20d5dbb05b94ab9ee6bb"
server_key_iv: "0b4f3476e308447eb0d70d16600a54c1"
server_key_version: "encryptionkey@0"
I20260928 21:54:56.532725 12818 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:54:56.533504 12832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:56.533668 12818 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 21:54:56.533748 12818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-2/wal
uuid: "12c9b4a5a5524132bfd9a35f43b634af"
format_stamp: "Formatted at 2026-09-28 21:54:56 on dist-test-slave-k6lz"
server_key: "9b645e0b72fd20d5dbb05b94ab9ee6bb"
server_key_iv: "0b4f3476e308447eb0d70d16600a54c1"
server_key_version: "encryptionkey@0"
I20260928 21:54:56.533828 12818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:56.537170 12818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:54:56.537406 12818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:54:56.537520 12818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:54:56.537700 12818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:54:56.537978 12818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:54:56.538010 12818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:56.538035 12818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:54:56.538053 12818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:56.543251 12818 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.67:40483
I20260928 21:54:56.543285 12945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.67:40483 every 8 connection(s)
I20260928 21:54:56.543594 12818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
I20260928 21:54:56.546327 12554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:56.544230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.67:51591 (local address 127.2.182.126:44425)
0928 21:54:56.544273 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:56.544275 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:56.544494 (+ 219us) server_negotiation.cc:407] Connection header received
0928 21:54:56.544971 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:56.544972 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:56.544979 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:56.545031 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:56.546098 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:56.546100 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:56.546100 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:54:56.546104 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:56.546118 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:54:56.546122 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:56.546170 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:56.546234 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 21:54:56.546247 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 21:54:56.546502 12948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:56.544200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:44425 (local address 127.2.182.67:51591)
0928 21:54:56.544386 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:56.544418 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:56.544869 (+ 451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:56.545101 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:56.545105 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:56.545114 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:56.545962 (+ 848us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:54:56.545963 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:56.545989 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:54:56.546016 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:56.546172 (+ 156us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:56.546176 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:56.546179 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:54:56.546243 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 21:54:56.546300 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":74,"threads_started":1}
I20260928 21:54:56.547077 12946 heartbeater.cc:347] Connected to a master server at 127.2.182.126:44425
I20260928 21:54:56.547153 12946 heartbeater.cc:464] Registering TS with master...
I20260928 21:54:56.547288 12946 heartbeater.cc:511] Master 127.2.182.126:44425 requested a full tablet report, sending...
I20260928 21:54:56.547647 12496 ts_manager.cc:194] Registered new tserver with Master: 12c9b4a5a5524132bfd9a35f43b634af (127.2.182.67:40483)
I20260928 21:54:56.552755 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 12818
I20260928 21:54:56.552834 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632440055392-2777-0/minicluster-data/ts-2/wal/instance
I20260928 21:54:56.552994 2777 external_mini_cluster.cc:1428] Setting key b14e742158d70afff19a71be81b4cc91
I20260928 21:54:56.553797 2777 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 21:54:56.555506 12554 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:54:56.555081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51660 (local address 127.2.182.126:44425)
0928 21:54:56.555133 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:56.555135 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:56.555220 (+ 85us) server_negotiation.cc:407] Connection header received
0928 21:54:56.555273 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:56.555274 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:56.555278 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:56.555323 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:56.555442 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51660: BlockingRecv error: recv got EOF from 127.0.0.1:51660 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 21:54:56.555608 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 12556
I20260928 21:54:56.560137 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 12687
I20260928 21:54:56.564118 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 12818
I20260928 21:54:56.567638 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 12465
2026-09-28T21:54:56Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (524 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:54:56 dist-test-slave-k6lz krb5kdc[12956](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:54:56 dist-test-slave-k6lz krb5kdc[12956](info): set up 2 sockets
Sep 28 21:54:56 dist-test-slave-k6lz krb5kdc[12956](info): commencing operation
krb5kdc: starting...
W20260928 21:54:57.524461 2777 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Sep 28 21:54:57 dist-test-slave-k6lz krb5kdc[12956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632497, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:54:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:54:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab.
Entry for principal kudu/127.2.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab.
I20260928 21:54:57.642226 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:40825
--webserver_interface=127.2.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab
--principal=kudu/127.2.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:33169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.126:40825
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:54:57.722218 12969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:54:57.722427 12969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:54:57.722450 12969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:54:57.723613 12969 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:54:57.723650 12969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:54:57.723663 12969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:54:57.723675 12969 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:54:57.723686 12969 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:54:57.725116 12969 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:33169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.126:40825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab
--principal=kudu/127.2.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:40825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:54:57.725310 12969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:54:57.725478 12969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:54:57.730304 12977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:57.730389 12974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:57.730436 12969 server_base.cc:1034] running on GCE node
W20260928 21:54:57.730588 12975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:57.730831 12969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:57.731142 12969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:57.732307 12969 hybrid_clock.cc:648] HybridClock initialized: now 1790632497732257 us; error 65 us; skew 500 ppm
Sep 28 21:54:57 dist-test-slave-k6lz krb5kdc[12956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632497, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:54:57.734803 12969 init.cc:377] Logged in from keytab as kudu/127.2.182.126@KRBTEST.COM (short username kudu)
I20260928 21:54:57.735219 12969 webserver.cc:460] Webserver started at http://127.2.182.126:34835/ using document root <none> and password file <none>
I20260928 21:54:57.735452 12969 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:57.735500 12969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:57.735618 12969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:54:57.736747 12969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/minicluster-data/master-0/data/instance:
uuid: "6d39899cabc64e7eab9167fca4edea5e"
format_stamp: "Formatted at 2026-09-28 21:54:57 on dist-test-slave-k6lz"
server_key: "fd1050c4df5bdcf7d80774106feb057d"
server_key_iv: "65021e525b022ed980ded04dcf0083c9"
server_key_version: "encryptionkey@0"
I20260928 21:54:57.737071 12969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/minicluster-data/master-0/wal/instance:
uuid: "6d39899cabc64e7eab9167fca4edea5e"
format_stamp: "Formatted at 2026-09-28 21:54:57 on dist-test-slave-k6lz"
server_key: "fd1050c4df5bdcf7d80774106feb057d"
server_key_iv: "65021e525b022ed980ded04dcf0083c9"
server_key_version: "encryptionkey@0"
I20260928 21:54:57.738373 12969 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:54:57.739210 12984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:57.739429 12969 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:54:57.739516 12969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/minicluster-data/master-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/minicluster-data/master-0/wal
uuid: "6d39899cabc64e7eab9167fca4edea5e"
format_stamp: "Formatted at 2026-09-28 21:54:57 on dist-test-slave-k6lz"
server_key: "fd1050c4df5bdcf7d80774106feb057d"
server_key_iv: "65021e525b022ed980ded04dcf0083c9"
server_key_version: "encryptionkey@0"
I20260928 21:54:57.739581 12969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:57.750809 12969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:54:57.752862 12969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:54:57.753001 12969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:54:57.756815 12969 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.126:40825
I20260928 21:54:57.756856 13036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.126:40825 every 8 connection(s)
I20260928 21:54:57.757175 12969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
I20260928 21:54:57.758137 13037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:54:57.760524 13037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d39899cabc64e7eab9167fca4edea5e: Bootstrap starting.
I20260928 21:54:57.761101 13037 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d39899cabc64e7eab9167fca4edea5e: Neither blocks nor log segments found. Creating new log.
I20260928 21:54:57.761368 13037 log.cc:824] T 00000000000000000000000000000000 P 6d39899cabc64e7eab9167fca4edea5e: Log is configured to *not* fsync() on all Append() calls
I20260928 21:54:57.762092 13037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d39899cabc64e7eab9167fca4edea5e: No bootstrap required, opened a new log
I20260928 21:54:57.763677 13037 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d39899cabc64e7eab9167fca4edea5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d39899cabc64e7eab9167fca4edea5e" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 40825 } }
I20260928 21:54:57.763816 13037 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d39899cabc64e7eab9167fca4edea5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:54:57.763837 13037 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d39899cabc64e7eab9167fca4edea5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d39899cabc64e7eab9167fca4edea5e, State: Initialized, Role: FOLLOWER
I20260928 21:54:57.763962 13037 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d39899cabc64e7eab9167fca4edea5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d39899cabc64e7eab9167fca4edea5e" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 40825 } }
I20260928 21:54:57.764029 13037 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d39899cabc64e7eab9167fca4edea5e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:54:57.764056 13037 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d39899cabc64e7eab9167fca4edea5e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:54:57.764102 13037 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d39899cabc64e7eab9167fca4edea5e [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:54:57.764854 13037 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d39899cabc64e7eab9167fca4edea5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d39899cabc64e7eab9167fca4edea5e" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 40825 } }
I20260928 21:54:57.764937 13037 leader_election.cc:302] T 00000000000000000000000000000000 P 6d39899cabc64e7eab9167fca4edea5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d39899cabc64e7eab9167fca4edea5e; no voters:
I20260928 21:54:57.765077 13037 leader_election.cc:288] T 00000000000000000000000000000000 P 6d39899cabc64e7eab9167fca4edea5e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:54:57.765153 13040 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d39899cabc64e7eab9167fca4edea5e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:54:57.765300 13037 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d39899cabc64e7eab9167fca4edea5e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:54:57.765328 13040 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d39899cabc64e7eab9167fca4edea5e [term 1 LEADER]: Becoming Leader. State: Replica: 6d39899cabc64e7eab9167fca4edea5e, State: Running, Role: LEADER
I20260928 21:54:57.765460 13040 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d39899cabc64e7eab9167fca4edea5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d39899cabc64e7eab9167fca4edea5e" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 40825 } }
I20260928 21:54:57.765908 13041 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d39899cabc64e7eab9167fca4edea5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d39899cabc64e7eab9167fca4edea5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d39899cabc64e7eab9167fca4edea5e" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 40825 } } }
I20260928 21:54:57.765983 13041 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d39899cabc64e7eab9167fca4edea5e [sys.catalog]: This master's current role is: LEADER
I20260928 21:54:57.766332 13042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d39899cabc64e7eab9167fca4edea5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d39899cabc64e7eab9167fca4edea5e. Latest consensus state: current_term: 1 leader_uuid: "6d39899cabc64e7eab9167fca4edea5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d39899cabc64e7eab9167fca4edea5e" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 40825 } } }
I20260928 21:54:57.766377 13049 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:54:57.766443 13042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d39899cabc64e7eab9167fca4edea5e [sys.catalog]: This master's current role is: LEADER
I20260928 21:54:57.766881 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 12969
I20260928 21:54:57.766969 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/minicluster-data/master-0/wal/instance
I20260928 21:54:57.767145 2777 external_mini_cluster.cc:1428] Setting key d73a7aeef571f6ddf22d5e3a45c12f57
I20260928 21:54:57.766916 13049 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:54:57.768999 13049 catalog_manager.cc:1347] Generated new cluster ID: b4cf91851be54ea097e31010510a68b0
I20260928 21:54:57.769053 13049 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 21:54:57 dist-test-slave-k6lz krb5kdc[12956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632497, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:54:57.778342 13059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:57.767621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51774 (local address 127.2.182.126:40825)
0928 21:54:57.767796 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:57.767806 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:57.767838 (+ 32us) server_negotiation.cc:407] Connection header received
0928 21:54:57.767988 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:57.767995 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:57.768052 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:57.768191 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:57.768904 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:57.770337 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:57.771389 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:57.772008 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:57.772172 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:57.774879 (+ 2707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:57.774883 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:57.774883 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:57.774895 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:57.776773 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:57.777110 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:57.777113 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:57.777113 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:57.777209 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:57.777375 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:57.777376 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:57.777376 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:57.777649 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:57.777759 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:57.777984 (+ 225us) server_negotiation.cc:299] Negotiation successful
0928 21:54:57.778117 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":79,"threads_started":1}
I20260928 21:54:57.798599 13049 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:54:57.799206 13049 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:54:57.802564 13049 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d39899cabc64e7eab9167fca4edea5e: Generated new TSK 0
I20260928 21:54:57.802739 13049 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.182.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab.
Entry for principal kudu/127.2.182.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab.
I20260928 21:54:57.830583 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.65:0
--local_ip_for_outbound_sockets=127.2.182.65
--webserver_interface=127.2.182.65
--webserver_port=0
--tserver_master_addrs=127.2.182.126:40825
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab
--principal=kudu/127.2.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:33169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:54:57.911080 13063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:54:57.911298 13063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:54:57.911360 13063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:54:57.912563 13063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:54:57.912608 13063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:54:57.912643 13063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.65
I20260928 21:54:57.914137 13063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:33169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab
--principal=kudu/127.2.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.182.65
--webserver_port=0
--tserver_master_addrs=127.2.182.126:40825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.65
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:54:57.914379 13063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:54:57.914590 13063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:54:57.919503 13068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:57.919538 13063 server_base.cc:1034] running on GCE node
W20260928 21:54:57.919539 13069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:57.919503 13071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:57.919874 13063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:57.920102 13063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:57.921245 13063 hybrid_clock.cc:648] HybridClock initialized: now 1790632497921228 us; error 36 us; skew 500 ppm
Sep 28 21:54:57 dist-test-slave-k6lz krb5kdc[12956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632497, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:54:57.923588 13063 init.cc:377] Logged in from keytab as kudu/127.2.182.65@KRBTEST.COM (short username kudu)
I20260928 21:54:57.924005 13063 webserver.cc:460] Webserver started at http://127.2.182.65:39469/ using document root <none> and password file <none>
I20260928 21:54:57.924216 13063 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:57.924263 13063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:57.924407 13063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:54:57.925421 13063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/minicluster-data/ts-0/data/instance:
uuid: "7386384cc1674d70a9e219fa0f9a95d2"
format_stamp: "Formatted at 2026-09-28 21:54:57 on dist-test-slave-k6lz"
server_key: "a74ba54ab423fdd4e2cd1c92313a2013"
server_key_iv: "0166cfb08b8ac6a7febd52d0066e6558"
server_key_version: "encryptionkey@0"
I20260928 21:54:57.925768 13063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/minicluster-data/ts-0/wal/instance:
uuid: "7386384cc1674d70a9e219fa0f9a95d2"
format_stamp: "Formatted at 2026-09-28 21:54:57 on dist-test-slave-k6lz"
server_key: "a74ba54ab423fdd4e2cd1c92313a2013"
server_key_iv: "0166cfb08b8ac6a7febd52d0066e6558"
server_key_version: "encryptionkey@0"
I20260928 21:54:57.927000 13063 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:54:57.927724 13078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:57.927913 13063 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:57.928012 13063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/minicluster-data/ts-0/wal
uuid: "7386384cc1674d70a9e219fa0f9a95d2"
format_stamp: "Formatted at 2026-09-28 21:54:57 on dist-test-slave-k6lz"
server_key: "a74ba54ab423fdd4e2cd1c92313a2013"
server_key_iv: "0166cfb08b8ac6a7febd52d0066e6558"
server_key_version: "encryptionkey@0"
I20260928 21:54:57.928073 13063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:57.939250 13063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:54:57.941545 13063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:54:57.941715 13063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:54:57.942030 13063 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:54:57.942386 13063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:54:57.942422 13063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:57.942451 13063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:54:57.942468 13063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:57.948014 13063 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.65:33153
I20260928 21:54:57.948057 13191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.65:33153 every 8 connection(s)
I20260928 21:54:57.948364 13063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
Sep 28 21:54:57 dist-test-slave-k6lz krb5kdc[12956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632497, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.65@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:54:57.955662 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 13063
I20260928 21:54:57.955770 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/minicluster-data/ts-0/wal/instance
I20260928 21:54:57.956101 2777 external_mini_cluster.cc:1428] Setting key 8d618f609e09d7fec8e736b81b100a39
I20260928 21:54:57.958393 13059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:57.949116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.65:36323 (local address 127.2.182.126:40825)
0928 21:54:57.949171 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:57.949173 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:57.949373 (+ 200us) server_negotiation.cc:407] Connection header received
0928 21:54:57.949881 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:57.949883 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:57.949897 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:57.949956 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:57.950943 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:57.951560 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:57.952638 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:57.952868 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:57.952927 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:57.955199 (+ 2272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:57.955201 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:57.955201 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:57.955208 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:57.956899 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:57.957292 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:57.957293 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:57.957293 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:57.957365 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:57.957628 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:57.957629 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:57.957629 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:57.957756 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:57.957818 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:57.958227 (+ 409us) server_negotiation.cc:299] Negotiation successful
0928 21:54:57.958316 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 21:54:57.958616 13194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:57.949094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:40825 (local address 127.2.182.65:36323)
0928 21:54:57.949255 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:57.949291 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:57.949759 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:57.950066 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:57.950069 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:57.950645 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:57.950842 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:57.950847 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:57.951646 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:57.951647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:57.952542 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:57.952544 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:57.952968 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:57.952969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:57.953079 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:57.953640 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:57.953648 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:57.955101 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:57.956970 (+ 1869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:57.956977 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:57.956979 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:57.957219 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:57.957451 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:57.957452 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:57.957453 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:57.957567 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:57.957841 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:57.957848 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:57.958067 (+ 219us) client_negotiation.cc:769] Sending connection context
0928 21:54:57.958193 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 21:54:57.958381 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":82,"threads_started":1}
I20260928 21:54:57.959170 13192 heartbeater.cc:347] Connected to a master server at 127.2.182.126:40825
I20260928 21:54:57.959264 13192 heartbeater.cc:464] Registering TS with master...
I20260928 21:54:57.959472 13192 heartbeater.cc:511] Master 127.2.182.126:40825 requested a full tablet report, sending...
I20260928 21:54:57.959985 13001 ts_manager.cc:194] Registered new tserver with Master: 7386384cc1674d70a9e219fa0f9a95d2 (127.2.182.65:33153)
I20260928 21:54:57.960664 13001 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.65@KRBTEST.COM'} at 127.2.182.65:36323
WARNING: no policy specified for kudu/127.2.182.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab.
Entry for principal kudu/127.2.182.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab.
I20260928 21:54:57.983404 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.66:0
--local_ip_for_outbound_sockets=127.2.182.66
--webserver_interface=127.2.182.66
--webserver_port=0
--tserver_master_addrs=127.2.182.126:40825
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab
--principal=kudu/127.2.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:33169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:54:58.064023 13197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:54:58.064231 13197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:54:58.064261 13197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:54:58.065375 13197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:54:58.065436 13197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:54:58.065475 13197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.66
I20260928 21:54:58.066967 13197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:33169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab
--principal=kudu/127.2.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.182.66
--webserver_port=0
--tserver_master_addrs=127.2.182.126:40825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.66
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:54:58.067198 13197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:54:58.067425 13197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:54:58.072108 13203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:58.072132 13202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:58.072188 13197 server_base.cc:1034] running on GCE node
W20260928 21:54:58.072144 13205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:58.072510 13197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:58.072751 13197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:58.073902 13197 hybrid_clock.cc:648] HybridClock initialized: now 1790632498073888 us; error 35 us; skew 500 ppm
Sep 28 21:54:58 dist-test-slave-k6lz krb5kdc[12956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632498, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:54:58.076380 13197 init.cc:377] Logged in from keytab as kudu/127.2.182.66@KRBTEST.COM (short username kudu)
I20260928 21:54:58.076804 13197 webserver.cc:460] Webserver started at http://127.2.182.66:40189/ using document root <none> and password file <none>
I20260928 21:54:58.076998 13197 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:58.077040 13197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:58.077155 13197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:54:58.078181 13197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/minicluster-data/ts-1/data/instance:
uuid: "d12b35630055487fa8ba2dc6c7f32683"
format_stamp: "Formatted at 2026-09-28 21:54:58 on dist-test-slave-k6lz"
server_key: "6920b4fcb8e7fd3de419ce872e5d3b15"
server_key_iv: "8e7f7015863c648dfe8dab2e781a3664"
server_key_version: "encryptionkey@0"
I20260928 21:54:58.078485 13197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/minicluster-data/ts-1/wal/instance:
uuid: "d12b35630055487fa8ba2dc6c7f32683"
format_stamp: "Formatted at 2026-09-28 21:54:58 on dist-test-slave-k6lz"
server_key: "6920b4fcb8e7fd3de419ce872e5d3b15"
server_key_iv: "8e7f7015863c648dfe8dab2e781a3664"
server_key_version: "encryptionkey@0"
I20260928 21:54:58.079712 13197 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:54:58.080574 13212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:58.080739 13197 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:54:58.080819 13197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/minicluster-data/ts-1/wal
uuid: "d12b35630055487fa8ba2dc6c7f32683"
format_stamp: "Formatted at 2026-09-28 21:54:58 on dist-test-slave-k6lz"
server_key: "6920b4fcb8e7fd3de419ce872e5d3b15"
server_key_iv: "8e7f7015863c648dfe8dab2e781a3664"
server_key_version: "encryptionkey@0"
I20260928 21:54:58.080885 13197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:58.108378 13197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:54:58.110966 13197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:54:58.111090 13197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:54:58.111374 13197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:54:58.111676 13197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:54:58.111709 13197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:58.111730 13197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:54:58.111742 13197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:58.118302 13197 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.66:33177
I20260928 21:54:58.118355 13325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.66:33177 every 8 connection(s)
I20260928 21:54:58.118690 13197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
Sep 28 21:54:58 dist-test-slave-k6lz krb5kdc[12956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632498, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.66@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:54:58.128690 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 13197
I20260928 21:54:58.128893 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/minicluster-data/ts-1/wal/instance
I20260928 21:54:58.129096 2777 external_mini_cluster.cc:1428] Setting key 430a9ed692cdd717ce33e4ad0477113f
I20260928 21:54:58.129168 13059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:58.119562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.66:57519 (local address 127.2.182.126:40825)
0928 21:54:58.119667 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:58.119669 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:58.119819 (+ 150us) server_negotiation.cc:407] Connection header received
0928 21:54:58.120389 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:58.120391 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:58.120403 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:58.120455 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:58.121978 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:58.122673 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:58.123769 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:58.123999 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:58.124057 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:58.126250 (+ 2193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:58.126252 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:58.126252 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:58.126259 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:58.127744 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:58.128181 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:58.128183 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:58.128183 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:58.128237 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:58.128512 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:58.128513 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:58.128514 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:58.128631 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:58.128689 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:58.128977 (+ 288us) server_negotiation.cc:299] Negotiation successful
0928 21:54:58.129077 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 21:54:58.129354 13328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:58.119479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:40825 (local address 127.2.182.66:57519)
0928 21:54:58.119693 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:58.119735 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:58.120270 (+ 535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:58.120535 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:58.120539 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:58.121614 (+ 1075us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:58.121839 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:58.121845 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:58.122788 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:58.122789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:58.123679 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:58.123683 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:58.124073 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:58.124074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:58.124200 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:58.124724 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:58.124733 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:58.126152 (+ 1419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:58.127831 (+ 1679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:58.127841 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:58.127845 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:58.128106 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:58.128339 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:58.128340 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:58.128340 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:58.128453 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:58.128697 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:58.128703 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:58.128845 (+ 142us) client_negotiation.cc:769] Sending connection context
0928 21:54:58.128949 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 21:54:58.129118 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":93,"threads_started":1}
I20260928 21:54:58.130056 13326 heartbeater.cc:347] Connected to a master server at 127.2.182.126:40825
I20260928 21:54:58.130174 13326 heartbeater.cc:464] Registering TS with master...
I20260928 21:54:58.130375 13326 heartbeater.cc:511] Master 127.2.182.126:40825 requested a full tablet report, sending...
I20260928 21:54:58.130779 13001 ts_manager.cc:194] Registered new tserver with Master: d12b35630055487fa8ba2dc6c7f32683 (127.2.182.66:33177)
I20260928 21:54:58.131475 13001 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.66@KRBTEST.COM'} at 127.2.182.66:57519
WARNING: no policy specified for kudu/127.2.182.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab.
Entry for principal kudu/127.2.182.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab.
I20260928 21:54:58.158360 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.67:0
--local_ip_for_outbound_sockets=127.2.182.67
--webserver_interface=127.2.182.67
--webserver_port=0
--tserver_master_addrs=127.2.182.126:40825
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab
--principal=kudu/127.2.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:33169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:54:58.239879 13331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:54:58.240094 13331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:54:58.240118 13331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:54:58.241317 13331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:54:58.241364 13331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:54:58.241390 13331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.67
I20260928 21:54:58.243006 13331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:33169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab
--principal=kudu/127.2.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.182.67
--webserver_port=0
--tserver_master_addrs=127.2.182.126:40825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.67
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:54:58.243234 13331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:54:58.243460 13331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:54:58.248657 13337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:54:58.248674 13336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:58.248764 13331 server_base.cc:1034] running on GCE node
W20260928 21:54:58.248669 13339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:54:58.249099 13331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:54:58.249306 13331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:54:58.250481 13331 hybrid_clock.cc:648] HybridClock initialized: now 1790632498250468 us; error 35 us; skew 500 ppm
Sep 28 21:54:58 dist-test-slave-k6lz krb5kdc[12956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632498, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:54:58.252996 13331 init.cc:377] Logged in from keytab as kudu/127.2.182.67@KRBTEST.COM (short username kudu)
I20260928 21:54:58.253400 13331 webserver.cc:460] Webserver started at http://127.2.182.67:39569/ using document root <none> and password file <none>
I20260928 21:54:58.253589 13331 fs_manager.cc:359] Metadata directory not provided
I20260928 21:54:58.253628 13331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:54:58.253715 13331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:54:58.254721 13331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/minicluster-data/ts-2/data/instance:
uuid: "a8eaf8dbdaad4e2c828334490f0d67a0"
format_stamp: "Formatted at 2026-09-28 21:54:58 on dist-test-slave-k6lz"
server_key: "7d26ffbfde629d2926e409a88138318a"
server_key_iv: "fd7972738016908b16d9d76a6aeb6ccc"
server_key_version: "encryptionkey@0"
I20260928 21:54:58.255010 13331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/minicluster-data/ts-2/wal/instance:
uuid: "a8eaf8dbdaad4e2c828334490f0d67a0"
format_stamp: "Formatted at 2026-09-28 21:54:58 on dist-test-slave-k6lz"
server_key: "7d26ffbfde629d2926e409a88138318a"
server_key_iv: "fd7972738016908b16d9d76a6aeb6ccc"
server_key_version: "encryptionkey@0"
I20260928 21:54:58.256315 13331 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 21:54:58.257172 13346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:58.257400 13331 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:54:58.257488 13331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/minicluster-data/ts-2/wal
uuid: "a8eaf8dbdaad4e2c828334490f0d67a0"
format_stamp: "Formatted at 2026-09-28 21:54:58 on dist-test-slave-k6lz"
server_key: "7d26ffbfde629d2926e409a88138318a"
server_key_iv: "fd7972738016908b16d9d76a6aeb6ccc"
server_key_version: "encryptionkey@0"
I20260928 21:54:58.257550 13331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:54:58.279606 13331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:54:58.281889 13331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:54:58.282004 13331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:54:58.282197 13331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:54:58.282451 13331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:54:58.282480 13331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:58.282501 13331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:54:58.282517 13331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:54:58.288086 13331 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.67:45491
I20260928 21:54:58.288149 13459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.67:45491 every 8 connection(s)
I20260928 21:54:58.288468 13331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
I20260928 21:54:58.293545 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 13331
I20260928 21:54:58.293630 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/minicluster-data/ts-2/wal/instance
I20260928 21:54:58.293864 2777 external_mini_cluster.cc:1428] Setting key 570cd595f448b7030cce2382ab121ba0
Sep 28 21:54:58 dist-test-slave-k6lz krb5kdc[12956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632498, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.67@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:54:58.298221 13059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:58.289301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.67:41539 (local address 127.2.182.126:40825)
0928 21:54:58.289395 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:58.289398 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:58.289587 (+ 189us) server_negotiation.cc:407] Connection header received
0928 21:54:58.290132 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:58.290134 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:58.290147 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:58.290197 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:58.291131 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:58.291657 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:58.292624 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:58.292809 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:58.292863 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:58.295157 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:58.295159 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:58.295159 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:58.295166 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:58.296686 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:58.297108 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:58.297109 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:58.297109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:58.297166 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:58.297530 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:58.297531 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:58.297531 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:58.297660 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:58.297731 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:58.298079 (+ 348us) server_negotiation.cc:299] Negotiation successful
0928 21:54:58.298153 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 21:54:58.298389 13462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:58.289208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:40825 (local address 127.2.182.67:41539)
0928 21:54:58.289412 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:58.289458 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:58.290011 (+ 553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:58.290263 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:58.290267 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:58.290826 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:54:58.291034 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:58.291039 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:58.291739 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:58.291739 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:58.292543 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:58.292546 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:58.292875 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:58.292876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:58.292999 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:58.293485 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:54:58.293493 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:54:58.295024 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:54:58.296776 (+ 1752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:58.296783 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:58.296785 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:58.297026 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:58.297252 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:54:58.297253 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:54:58.297254 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:54:58.297426 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:54:58.297750 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:54:58.297756 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:54:58.297924 (+ 168us) client_negotiation.cc:769] Sending connection context
0928 21:54:58.298019 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 21:54:58.298161 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":92,"threads_started":1}
I20260928 21:54:58.299188 13460 heartbeater.cc:347] Connected to a master server at 127.2.182.126:40825
I20260928 21:54:58.299505 13460 heartbeater.cc:464] Registering TS with master...
I20260928 21:54:58.299731 13460 heartbeater.cc:511] Master 127.2.182.126:40825 requested a full tablet report, sending...
I20260928 21:54:58.300168 13001 ts_manager.cc:194] Registered new tserver with Master: a8eaf8dbdaad4e2c828334490f0d67a0 (127.2.182.67:45491)
I20260928 21:54:58.300573 13001 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.67@KRBTEST.COM'} at 127.2.182.67:41539
I20260928 21:54:58.305846 2777 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:54:58.313218 13059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:58.307188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51790 (local address 127.2.182.126:40825)
0928 21:54:58.307270 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:58.307272 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:58.307354 (+ 82us) server_negotiation.cc:407] Connection header received
0928 21:54:58.307427 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:58.307428 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:58.307438 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:58.307522 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:54:58.308127 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:58.308621 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:58.309301 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:58.309509 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:58.309562 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:58.310815 (+ 1253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:54:58.310816 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:54:58.310817 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:54:58.310824 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:54:58.312181 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:58.312488 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:58.312489 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:58.312490 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:58.312540 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:54:58.312701 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:54:58.312702 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:54:58.312702 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:54:58.312853 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:54:58.312947 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:58.313058 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 21:54:58.313130 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 21:54:58.314836 13001 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51790:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:54:58.316547 13001 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:54:58.322950 13473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:58.318853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49918 (local address 127.2.182.67:45491)
0928 21:54:58.319440 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:58.319444 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:58.319452 (+ 8us) server_negotiation.cc:407] Connection header received
0928 21:54:58.319484 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:58.319486 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:58.319500 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:58.319771 (+ 271us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:54:58.319845 (+ 74us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:58.320851 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:58.322100 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:58.322582 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:58.322644 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:58.322705 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:58.322829 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 21:54:58.322869 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":99,"threads_started":1}
I20260928 21:54:58.322942 13474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:58.318754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.67:45491 (local address 127.0.0.1:49918)
0928 21:54:58.319210 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:58.319219 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:58.319269 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:58.319600 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:58.319601 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:58.319603 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:54:58.319658 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:58.319659 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:58.321001 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:58.321002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:58.322034 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:58.322036 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:58.322657 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:58.322658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:58.322734 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:58.322747 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 21:54:58.322809 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 21:54:58.322859 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"mutex_wait_us":2,"thread_start_us":53,"threads_started":1}
I20260928 21:54:58.323083 13475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:58.318930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57050 (local address 127.2.182.66:33177)
0928 21:54:58.319093 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:58.319096 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:58.319240 (+ 144us) server_negotiation.cc:407] Connection header received
0928 21:54:58.319323 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:58.319325 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:58.319361 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:58.319439 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:54:58.319621 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:58.320797 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:58.321889 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:58.322637 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:58.322704 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:58.322757 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:58.322966 (+ 209us) server_negotiation.cc:299] Negotiation successful
0928 21:54:58.323009 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":75,"threads_started":1}
I20260928 21:54:58.323099 13476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:58.319055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56530 (local address 127.2.182.65:33153)
0928 21:54:58.319205 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:58.319210 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:58.319538 (+ 328us) server_negotiation.cc:407] Connection header received
0928 21:54:58.319580 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:58.319583 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:58.319598 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:58.319661 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:54:58.319843 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:58.320541 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:58.321924 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:58.322636 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:58.322704 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:58.322757 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:58.322995 (+ 238us) server_negotiation.cc:299] Negotiation successful
0928 21:54:58.323034 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":70,"threads_started":1}
I20260928 21:54:58.323279 13471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:58.318754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.66:33177 (local address 127.0.0.1:57050)
0928 21:54:58.319140 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:58.319170 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:58.319238 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:58.319450 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:58.319452 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:58.319455 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:54:58.319529 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:58.319531 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:58.320917 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:58.320918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:58.321816 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:58.321818 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:58.322791 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:58.322793 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:58.322861 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:58.322868 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 21:54:58.323171 (+ 303us) client_negotiation.cc:240] Negotiation successful
0928 21:54:58.323213 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"mutex_wait_us":23,"thread_start_us":73,"threads_started":1}
I20260928 21:54:58.323463 13472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:58.318936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.65:33153 (local address 127.0.0.1:56530)
0928 21:54:58.319377 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:58.319387 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:58.319475 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:58.319736 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:58.319736 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:58.319738 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:54:58.319769 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:58.319770 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:58.320634 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:58.320635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:58.321842 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:58.321844 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:58.322793 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:58.322794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:58.322861 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:58.322867 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 21:54:58.322911 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 21:54:58.322955 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"mutex_wait_us":4,"thread_start_us":68,"threads_started":1}
I20260928 21:54:58.324091 13260 tablet_service.cc:1467] Processing CreateTablet for tablet 33e583fc9d3e4b738b4f80342c8f4da2 (DEFAULT_TABLE table=test-table [id=82490f257de54633b2b777b8681a42ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:54:58.324081 13394 tablet_service.cc:1467] Processing CreateTablet for tablet 33e583fc9d3e4b738b4f80342c8f4da2 (DEFAULT_TABLE table=test-table [id=82490f257de54633b2b777b8681a42ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:54:58.324189 13126 tablet_service.cc:1467] Processing CreateTablet for tablet 33e583fc9d3e4b738b4f80342c8f4da2 (DEFAULT_TABLE table=test-table [id=82490f257de54633b2b777b8681a42ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:54:58.324431 13260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33e583fc9d3e4b738b4f80342c8f4da2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:54:58.324431 13394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33e583fc9d3e4b738b4f80342c8f4da2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:54:58.324460 13126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33e583fc9d3e4b738b4f80342c8f4da2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:54:58.326968 13477 tablet_bootstrap.cc:492] T 33e583fc9d3e4b738b4f80342c8f4da2 P d12b35630055487fa8ba2dc6c7f32683: Bootstrap starting.
I20260928 21:54:58.327487 13479 tablet_bootstrap.cc:492] T 33e583fc9d3e4b738b4f80342c8f4da2 P 7386384cc1674d70a9e219fa0f9a95d2: Bootstrap starting.
I20260928 21:54:58.327706 13477 tablet_bootstrap.cc:654] T 33e583fc9d3e4b738b4f80342c8f4da2 P d12b35630055487fa8ba2dc6c7f32683: Neither blocks nor log segments found. Creating new log.
I20260928 21:54:58.328107 13479 tablet_bootstrap.cc:654] T 33e583fc9d3e4b738b4f80342c8f4da2 P 7386384cc1674d70a9e219fa0f9a95d2: Neither blocks nor log segments found. Creating new log.
I20260928 21:54:58.328265 13477 log.cc:824] T 33e583fc9d3e4b738b4f80342c8f4da2 P d12b35630055487fa8ba2dc6c7f32683: Log is configured to *not* fsync() on all Append() calls
I20260928 21:54:58.328644 13479 log.cc:824] T 33e583fc9d3e4b738b4f80342c8f4da2 P 7386384cc1674d70a9e219fa0f9a95d2: Log is configured to *not* fsync() on all Append() calls
I20260928 21:54:58.329011 13477 tablet_bootstrap.cc:492] T 33e583fc9d3e4b738b4f80342c8f4da2 P d12b35630055487fa8ba2dc6c7f32683: No bootstrap required, opened a new log
I20260928 21:54:58.329078 13477 ts_tablet_manager.cc:1397] T 33e583fc9d3e4b738b4f80342c8f4da2 P d12b35630055487fa8ba2dc6c7f32683: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 21:54:58.329300 13479 tablet_bootstrap.cc:492] T 33e583fc9d3e4b738b4f80342c8f4da2 P 7386384cc1674d70a9e219fa0f9a95d2: No bootstrap required, opened a new log
I20260928 21:54:58.329363 13479 ts_tablet_manager.cc:1397] T 33e583fc9d3e4b738b4f80342c8f4da2 P 7386384cc1674d70a9e219fa0f9a95d2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:54:58.331067 13477 raft_consensus.cc:348] T 33e583fc9d3e4b738b4f80342c8f4da2 P d12b35630055487fa8ba2dc6c7f32683 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d12b35630055487fa8ba2dc6c7f32683" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 33177 } } peers { permanent_uuid: "7386384cc1674d70a9e219fa0f9a95d2" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 33153 } } peers { permanent_uuid: "a8eaf8dbdaad4e2c828334490f0d67a0" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 45491 } }
I20260928 21:54:58.331246 13477 raft_consensus.cc:374] T 33e583fc9d3e4b738b4f80342c8f4da2 P d12b35630055487fa8ba2dc6c7f32683 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:54:58.331281 13477 raft_consensus.cc:729] T 33e583fc9d3e4b738b4f80342c8f4da2 P d12b35630055487fa8ba2dc6c7f32683 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d12b35630055487fa8ba2dc6c7f32683, State: Initialized, Role: FOLLOWER
I20260928 21:54:58.331203 13479 raft_consensus.cc:348] T 33e583fc9d3e4b738b4f80342c8f4da2 P 7386384cc1674d70a9e219fa0f9a95d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d12b35630055487fa8ba2dc6c7f32683" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 33177 } } peers { permanent_uuid: "7386384cc1674d70a9e219fa0f9a95d2" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 33153 } } peers { permanent_uuid: "a8eaf8dbdaad4e2c828334490f0d67a0" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 45491 } }
I20260928 21:54:58.331347 13479 raft_consensus.cc:374] T 33e583fc9d3e4b738b4f80342c8f4da2 P 7386384cc1674d70a9e219fa0f9a95d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:54:58.331387 13479 raft_consensus.cc:729] T 33e583fc9d3e4b738b4f80342c8f4da2 P 7386384cc1674d70a9e219fa0f9a95d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7386384cc1674d70a9e219fa0f9a95d2, State: Initialized, Role: FOLLOWER
I20260928 21:54:58.331481 13479 consensus_queue.cc:260] T 33e583fc9d3e4b738b4f80342c8f4da2 P 7386384cc1674d70a9e219fa0f9a95d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d12b35630055487fa8ba2dc6c7f32683" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 33177 } } peers { permanent_uuid: "7386384cc1674d70a9e219fa0f9a95d2" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 33153 } } peers { permanent_uuid: "a8eaf8dbdaad4e2c828334490f0d67a0" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 45491 } }
I20260928 21:54:58.331468 13477 consensus_queue.cc:260] T 33e583fc9d3e4b738b4f80342c8f4da2 P d12b35630055487fa8ba2dc6c7f32683 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d12b35630055487fa8ba2dc6c7f32683" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 33177 } } peers { permanent_uuid: "7386384cc1674d70a9e219fa0f9a95d2" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 33153 } } peers { permanent_uuid: "a8eaf8dbdaad4e2c828334490f0d67a0" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 45491 } }
I20260928 21:54:58.331749 13477 ts_tablet_manager.cc:1428] T 33e583fc9d3e4b738b4f80342c8f4da2 P d12b35630055487fa8ba2dc6c7f32683: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260928 21:54:58.331749 13479 ts_tablet_manager.cc:1428] T 33e583fc9d3e4b738b4f80342c8f4da2 P 7386384cc1674d70a9e219fa0f9a95d2: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 21:54:58.331754 13192 heartbeater.cc:503] Master 127.2.182.126:40825 was elected leader, sending a full tablet report...
I20260928 21:54:58.328975 13478 tablet_bootstrap.cc:492] T 33e583fc9d3e4b738b4f80342c8f4da2 P a8eaf8dbdaad4e2c828334490f0d67a0: Bootstrap starting.
I20260928 21:54:58.331876 13326 heartbeater.cc:503] Master 127.2.182.126:40825 was elected leader, sending a full tablet report...
I20260928 21:54:58.332392 13478 tablet_bootstrap.cc:654] T 33e583fc9d3e4b738b4f80342c8f4da2 P a8eaf8dbdaad4e2c828334490f0d67a0: Neither blocks nor log segments found. Creating new log.
I20260928 21:54:58.332887 13478 log.cc:824] T 33e583fc9d3e4b738b4f80342c8f4da2 P a8eaf8dbdaad4e2c828334490f0d67a0: Log is configured to *not* fsync() on all Append() calls
I20260928 21:54:58.333642 13478 tablet_bootstrap.cc:492] T 33e583fc9d3e4b738b4f80342c8f4da2 P a8eaf8dbdaad4e2c828334490f0d67a0: No bootstrap required, opened a new log
I20260928 21:54:58.333714 13478 ts_tablet_manager.cc:1397] T 33e583fc9d3e4b738b4f80342c8f4da2 P a8eaf8dbdaad4e2c828334490f0d67a0: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260928 21:54:58.335129 13478 raft_consensus.cc:348] T 33e583fc9d3e4b738b4f80342c8f4da2 P a8eaf8dbdaad4e2c828334490f0d67a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d12b35630055487fa8ba2dc6c7f32683" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 33177 } } peers { permanent_uuid: "7386384cc1674d70a9e219fa0f9a95d2" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 33153 } } peers { permanent_uuid: "a8eaf8dbdaad4e2c828334490f0d67a0" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 45491 } }
I20260928 21:54:58.335266 13478 raft_consensus.cc:374] T 33e583fc9d3e4b738b4f80342c8f4da2 P a8eaf8dbdaad4e2c828334490f0d67a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:54:58.335296 13478 raft_consensus.cc:729] T 33e583fc9d3e4b738b4f80342c8f4da2 P a8eaf8dbdaad4e2c828334490f0d67a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8eaf8dbdaad4e2c828334490f0d67a0, State: Initialized, Role: FOLLOWER
I20260928 21:54:58.335443 13478 consensus_queue.cc:260] T 33e583fc9d3e4b738b4f80342c8f4da2 P a8eaf8dbdaad4e2c828334490f0d67a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d12b35630055487fa8ba2dc6c7f32683" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 33177 } } peers { permanent_uuid: "7386384cc1674d70a9e219fa0f9a95d2" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 33153 } } peers { permanent_uuid: "a8eaf8dbdaad4e2c828334490f0d67a0" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 45491 } }
I20260928 21:54:58.335696 13478 ts_tablet_manager.cc:1428] T 33e583fc9d3e4b738b4f80342c8f4da2 P a8eaf8dbdaad4e2c828334490f0d67a0: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 21:54:58.335762 13460 heartbeater.cc:503] Master 127.2.182.126:40825 was elected leader, sending a full tablet report...
W20260928 21:54:58.369205 13327 tablet.cc:2369] T 33e583fc9d3e4b738b4f80342c8f4da2 P d12b35630055487fa8ba2dc6c7f32683: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 21:54:58.449101 13193 tablet.cc:2369] T 33e583fc9d3e4b738b4f80342c8f4da2 P 7386384cc1674d70a9e219fa0f9a95d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:54:58.485052 13483 raft_consensus.cc:482] T 33e583fc9d3e4b738b4f80342c8f4da2 P d12b35630055487fa8ba2dc6c7f32683 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:54:58.485203 13483 raft_consensus.cc:504] T 33e583fc9d3e4b738b4f80342c8f4da2 P d12b35630055487fa8ba2dc6c7f32683 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d12b35630055487fa8ba2dc6c7f32683" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 33177 } } peers { permanent_uuid: "7386384cc1674d70a9e219fa0f9a95d2" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 33153 } } peers { permanent_uuid: "a8eaf8dbdaad4e2c828334490f0d67a0" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 45491 } }
I20260928 21:54:58.485600 13483 leader_election.cc:288] T 33e583fc9d3e4b738b4f80342c8f4da2 P d12b35630055487fa8ba2dc6c7f32683 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7386384cc1674d70a9e219fa0f9a95d2 (127.2.182.65:33153), a8eaf8dbdaad4e2c828334490f0d67a0 (127.2.182.67:45491)
I20260928 21:54:58.488538 13476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:58.486032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.66:33201 (local address 127.2.182.65:33153)
0928 21:54:58.486100 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:58.486102 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:58.486113 (+ 11us) server_negotiation.cc:407] Connection header received
0928 21:54:58.486143 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:58.486144 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:58.486153 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:58.486215 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:54:58.486355 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:58.486901 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:58.487863 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:58.488222 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:58.488283 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:58.488292 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:58.488441 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 21:54:58.488475 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 21:54:58.488538 13328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:58.485778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.65:33153 (local address 127.2.182.66:33201)
0928 21:54:58.485886 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:58.485897 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:58.485958 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:58.486228 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:58.486229 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:58.486231 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:54:58.486301 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:58.486303 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:58.486984 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:58.486985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:58.487780 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:58.487783 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:58.488285 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:58.488286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:58.488373 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:58.488376 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:54:58.488419 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 21:54:58.488460 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260928 21:54:58.488835 13473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:58.485852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.66:58961 (local address 127.2.182.67:45491)
0928 21:54:58.486083 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:58.486086 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:58.486620 (+ 534us) server_negotiation.cc:407] Connection header received
0928 21:54:58.486648 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:58.486649 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:58.486660 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:58.486700 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:54:58.486833 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:58.487283 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:58.488161 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:58.488543 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:58.488658 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:58.488667 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:58.488738 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 21:54:58.488775 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209}
I20260928 21:54:58.488850 13486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:58.485778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.67:45491 (local address 127.2.182.66:58961)
0928 21:54:58.486553 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:58.486561 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 21:54:58.486609 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:54:58.486713 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:54:58.486714 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:54:58.486715 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:54:58.486774 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:58.486775 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:58.487370 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:58.487371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:58.488082 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:54:58.488083 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:54:58.488618 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:54:58.488619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:54:58.488684 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:58.488688 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:54:58.488734 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 21:54:58.488771 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":761,"mutex_wait_us":101,"thread_start_us":174,"threads_started":1}
I20260928 21:54:58.488883 13146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e583fc9d3e4b738b4f80342c8f4da2" candidate_uuid: "d12b35630055487fa8ba2dc6c7f32683" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7386384cc1674d70a9e219fa0f9a95d2" is_pre_election: true
I20260928 21:54:58.489053 13146 raft_consensus.cc:2454] T 33e583fc9d3e4b738b4f80342c8f4da2 P 7386384cc1674d70a9e219fa0f9a95d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d12b35630055487fa8ba2dc6c7f32683 in term 0.
I20260928 21:54:58.489287 13213 leader_election.cc:302] T 33e583fc9d3e4b738b4f80342c8f4da2 P d12b35630055487fa8ba2dc6c7f32683 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7386384cc1674d70a9e219fa0f9a95d2, d12b35630055487fa8ba2dc6c7f32683; no voters:
I20260928 21:54:58.489429 13483 raft_consensus.cc:2781] T 33e583fc9d3e4b738b4f80342c8f4da2 P d12b35630055487fa8ba2dc6c7f32683 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:54:58.489408 13414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e583fc9d3e4b738b4f80342c8f4da2" candidate_uuid: "d12b35630055487fa8ba2dc6c7f32683" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8eaf8dbdaad4e2c828334490f0d67a0" is_pre_election: true
I20260928 21:54:58.489477 13483 raft_consensus.cc:482] T 33e583fc9d3e4b738b4f80342c8f4da2 P d12b35630055487fa8ba2dc6c7f32683 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:54:58.489496 13483 raft_consensus.cc:3037] T 33e583fc9d3e4b738b4f80342c8f4da2 P d12b35630055487fa8ba2dc6c7f32683 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:54:58.489543 13414 raft_consensus.cc:2454] T 33e583fc9d3e4b738b4f80342c8f4da2 P a8eaf8dbdaad4e2c828334490f0d67a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d12b35630055487fa8ba2dc6c7f32683 in term 0.
I20260928 21:54:58.490418 13483 raft_consensus.cc:504] T 33e583fc9d3e4b738b4f80342c8f4da2 P d12b35630055487fa8ba2dc6c7f32683 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d12b35630055487fa8ba2dc6c7f32683" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 33177 } } peers { permanent_uuid: "7386384cc1674d70a9e219fa0f9a95d2" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 33153 } } peers { permanent_uuid: "a8eaf8dbdaad4e2c828334490f0d67a0" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 45491 } }
I20260928 21:54:58.490567 13483 leader_election.cc:288] T 33e583fc9d3e4b738b4f80342c8f4da2 P d12b35630055487fa8ba2dc6c7f32683 [CANDIDATE]: Term 1 election: Requested vote from peers 7386384cc1674d70a9e219fa0f9a95d2 (127.2.182.65:33153), a8eaf8dbdaad4e2c828334490f0d67a0 (127.2.182.67:45491)
I20260928 21:54:58.490749 13414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e583fc9d3e4b738b4f80342c8f4da2" candidate_uuid: "d12b35630055487fa8ba2dc6c7f32683" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8eaf8dbdaad4e2c828334490f0d67a0"
I20260928 21:54:58.490758 13146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e583fc9d3e4b738b4f80342c8f4da2" candidate_uuid: "d12b35630055487fa8ba2dc6c7f32683" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7386384cc1674d70a9e219fa0f9a95d2"
I20260928 21:54:58.490831 13414 raft_consensus.cc:3037] T 33e583fc9d3e4b738b4f80342c8f4da2 P a8eaf8dbdaad4e2c828334490f0d67a0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:54:58.490831 13146 raft_consensus.cc:3037] T 33e583fc9d3e4b738b4f80342c8f4da2 P 7386384cc1674d70a9e219fa0f9a95d2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:54:58.491726 13146 raft_consensus.cc:2454] T 33e583fc9d3e4b738b4f80342c8f4da2 P 7386384cc1674d70a9e219fa0f9a95d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d12b35630055487fa8ba2dc6c7f32683 in term 1.
I20260928 21:54:58.491726 13414 raft_consensus.cc:2454] T 33e583fc9d3e4b738b4f80342c8f4da2 P a8eaf8dbdaad4e2c828334490f0d67a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d12b35630055487fa8ba2dc6c7f32683 in term 1.
I20260928 21:54:58.491916 13213 leader_election.cc:302] T 33e583fc9d3e4b738b4f80342c8f4da2 P d12b35630055487fa8ba2dc6c7f32683 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7386384cc1674d70a9e219fa0f9a95d2, d12b35630055487fa8ba2dc6c7f32683; no voters:
I20260928 21:54:58.492040 13483 raft_consensus.cc:2781] T 33e583fc9d3e4b738b4f80342c8f4da2 P d12b35630055487fa8ba2dc6c7f32683 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:54:58.492156 13483 raft_consensus.cc:686] T 33e583fc9d3e4b738b4f80342c8f4da2 P d12b35630055487fa8ba2dc6c7f32683 [term 1 LEADER]: Becoming Leader. State: Replica: d12b35630055487fa8ba2dc6c7f32683, State: Running, Role: LEADER
I20260928 21:54:58.492231 13483 consensus_queue.cc:237] T 33e583fc9d3e4b738b4f80342c8f4da2 P d12b35630055487fa8ba2dc6c7f32683 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d12b35630055487fa8ba2dc6c7f32683" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 33177 } } peers { permanent_uuid: "7386384cc1674d70a9e219fa0f9a95d2" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 33153 } } peers { permanent_uuid: "a8eaf8dbdaad4e2c828334490f0d67a0" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 45491 } }
I20260928 21:54:58.493033 13000 catalog_manager.cc:5495] T 33e583fc9d3e4b738b4f80342c8f4da2 P d12b35630055487fa8ba2dc6c7f32683 reported cstate change: term changed from 0 to 1, leader changed from <none> to d12b35630055487fa8ba2dc6c7f32683 (127.2.182.66). New cstate: current_term: 1 leader_uuid: "d12b35630055487fa8ba2dc6c7f32683" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d12b35630055487fa8ba2dc6c7f32683" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 33177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7386384cc1674d70a9e219fa0f9a95d2" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 33153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8eaf8dbdaad4e2c828334490f0d67a0" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 45491 } health_report { overall_health: UNKNOWN } } }
I20260928 21:54:58.508517 13475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:58.506110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57056 (local address 127.2.182.66:33177)
0928 21:54:58.506169 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 21:54:58.506171 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:54:58.506180 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:54:58.506252 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:54:58.506253 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:54:58.506271 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:54:58.506337 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:54:58.506462 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:58.506963 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:58.507806 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:54:58.507999 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:54:58.508052 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:54:58.508190 (+ 138us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:54:58.508291 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:54:58.508361 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 21:54:58.508399 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 21:54:58.508420 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 21:54:58.510223 13414 raft_consensus.cc:1261] T 33e583fc9d3e4b738b4f80342c8f4da2 P a8eaf8dbdaad4e2c828334490f0d67a0 [term 1 FOLLOWER]: Refusing update from remote peer d12b35630055487fa8ba2dc6c7f32683: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:54:58.510250 13146 raft_consensus.cc:1261] T 33e583fc9d3e4b738b4f80342c8f4da2 P 7386384cc1674d70a9e219fa0f9a95d2 [term 1 FOLLOWER]: Refusing update from remote peer d12b35630055487fa8ba2dc6c7f32683: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:54:58.510564 13483 consensus_queue.cc:1035] T 33e583fc9d3e4b738b4f80342c8f4da2 P d12b35630055487fa8ba2dc6c7f32683 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7386384cc1674d70a9e219fa0f9a95d2" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 33153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:54:58.510705 13487 consensus_queue.cc:1035] T 33e583fc9d3e4b738b4f80342c8f4da2 P d12b35630055487fa8ba2dc6c7f32683 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8eaf8dbdaad4e2c828334490f0d67a0" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 45491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:54:58.513872 13492 mvcc.cc:205] Tried to move back new op lower bound from 7334430713895317504 to 7334430713824739328. Current Snapshot: MvccSnapshot[applied={T|T < 7334430713895317504}]
I20260928 21:54:58.515312 13001 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51790:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:54:58.515439 13001 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51790:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:54:58.516144 13001 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6d39899cabc64e7eab9167fca4edea5e: Sending DeleteTablet for 3 replicas of tablet 33e583fc9d3e4b738b4f80342c8f4da2
I20260928 21:54:58.516440 13394 tablet_service.cc:1514] Processing DeleteTablet for tablet 33e583fc9d3e4b738b4f80342c8f4da2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:54:58 UTC) from {username='kudu', principal='kudu/127.2.182.126@KRBTEST.COM'} at 127.0.0.1:49918
I20260928 21:54:58.516505 13260 tablet_service.cc:1514] Processing DeleteTablet for tablet 33e583fc9d3e4b738b4f80342c8f4da2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:54:58 UTC) from {username='kudu', principal='kudu/127.2.182.126@KRBTEST.COM'} at 127.0.0.1:57050
I20260928 21:54:58.516536 13126 tablet_service.cc:1514] Processing DeleteTablet for tablet 33e583fc9d3e4b738b4f80342c8f4da2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:54:58 UTC) from {username='kudu', principal='kudu/127.2.182.126@KRBTEST.COM'} at 127.0.0.1:56530
I20260928 21:54:58.516709 13501 tablet_replica.cc:330] T 33e583fc9d3e4b738b4f80342c8f4da2 P 7386384cc1674d70a9e219fa0f9a95d2: stopping tablet replica
I20260928 21:54:58.516767 13499 tablet_replica.cc:330] T 33e583fc9d3e4b738b4f80342c8f4da2 P a8eaf8dbdaad4e2c828334490f0d67a0: stopping tablet replica
I20260928 21:54:58.516808 13501 raft_consensus.cc:2229] T 33e583fc9d3e4b738b4f80342c8f4da2 P 7386384cc1674d70a9e219fa0f9a95d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:54:58.516839 13499 raft_consensus.cc:2229] T 33e583fc9d3e4b738b4f80342c8f4da2 P a8eaf8dbdaad4e2c828334490f0d67a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:54:58.516914 13501 raft_consensus.cc:2258] T 33e583fc9d3e4b738b4f80342c8f4da2 P 7386384cc1674d70a9e219fa0f9a95d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:54:58.516916 13499 raft_consensus.cc:2258] T 33e583fc9d3e4b738b4f80342c8f4da2 P a8eaf8dbdaad4e2c828334490f0d67a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:54:58.517019 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 13063
I20260928 21:54:58.517565 13500 tablet_replica.cc:330] T 33e583fc9d3e4b738b4f80342c8f4da2 P d12b35630055487fa8ba2dc6c7f32683: stopping tablet replica
I20260928 21:54:58.517827 13500 raft_consensus.cc:2229] T 33e583fc9d3e4b738b4f80342c8f4da2 P d12b35630055487fa8ba2dc6c7f32683 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:54:58.517918 13500 raft_consensus.cc:2258] T 33e583fc9d3e4b738b4f80342c8f4da2 P d12b35630055487fa8ba2dc6c7f32683 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:54:58.517915 13499 ts_tablet_manager.cc:1905] T 33e583fc9d3e4b738b4f80342c8f4da2 P a8eaf8dbdaad4e2c828334490f0d67a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:54:58.518340 13500 ts_tablet_manager.cc:1905] T 33e583fc9d3e4b738b4f80342c8f4da2 P d12b35630055487fa8ba2dc6c7f32683: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:54:58.519610 13499 ts_tablet_manager.cc:1918] T 33e583fc9d3e4b738b4f80342c8f4da2 P a8eaf8dbdaad4e2c828334490f0d67a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:54:58.519675 13499 log.cc:1196] T 33e583fc9d3e4b738b4f80342c8f4da2 P a8eaf8dbdaad4e2c828334490f0d67a0: Deleting WAL directory at /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/minicluster-data/ts-2/wal/wals/33e583fc9d3e4b738b4f80342c8f4da2
I20260928 21:54:58.519912 13500 ts_tablet_manager.cc:1918] T 33e583fc9d3e4b738b4f80342c8f4da2 P d12b35630055487fa8ba2dc6c7f32683: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:54:58.519959 13499 ts_tablet_manager.cc:1939] T 33e583fc9d3e4b738b4f80342c8f4da2 P a8eaf8dbdaad4e2c828334490f0d67a0: Deleting consensus metadata
I20260928 21:54:58.519968 13500 log.cc:1196] T 33e583fc9d3e4b738b4f80342c8f4da2 P d12b35630055487fa8ba2dc6c7f32683: Deleting WAL directory at /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632440055392-2777-0/minicluster-data/ts-1/wal/wals/33e583fc9d3e4b738b4f80342c8f4da2
I20260928 21:54:58.520198 13500 ts_tablet_manager.cc:1939] T 33e583fc9d3e4b738b4f80342c8f4da2 P d12b35630055487fa8ba2dc6c7f32683: Deleting consensus metadata
I20260928 21:54:58.520462 12986 catalog_manager.cc:4841] TS a8eaf8dbdaad4e2c828334490f0d67a0 (127.2.182.67:45491): tablet 33e583fc9d3e4b738b4f80342c8f4da2 (table test-table [id=82490f257de54633b2b777b8681a42ff]) successfully deleted
I20260928 21:54:58.520557 12985 catalog_manager.cc:4841] TS d12b35630055487fa8ba2dc6c7f32683 (127.2.182.66:33177): tablet 33e583fc9d3e4b738b4f80342c8f4da2 (table test-table [id=82490f257de54633b2b777b8681a42ff]) successfully deleted
W20260928 21:54:58.524199 12985 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.182.65:33153 (error 108)
I20260928 21:54:58.524437 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 13197
I20260928 21:54:58.525101 13471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:54:58.524614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.65:33153 (local address 127.0.0.1:56532)
0928 21:54:58.525018 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0928 21:54:58.525056 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.65:33153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383}
W20260928 21:54:58.526327 12985 catalog_manager.cc:4584] TS 7386384cc1674d70a9e219fa0f9a95d2 (127.2.182.65:33153): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 33e583fc9d3e4b738b4f80342c8f4da2: Network error: Client connection negotiation failed: client connection to 127.2.182.65:33153: connect: Connection refused (error 111)
I20260928 21:54:58.529188 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 13331
I20260928 21:54:58.533353 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 12969
2026-09-28T21:54:58Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1966 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:54:58 dist-test-slave-k6lz krb5kdc[13505](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:54:58 dist-test-slave-k6lz krb5kdc[13505](info): set up 2 sockets
Sep 28 21:54:58 dist-test-slave-k6lz krb5kdc[13505](info): commencing operation
krb5kdc: starting...
W20260928 21:54:59.492057 2777 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Sep 28 21:54:59 dist-test-slave-k6lz krb5kdc[13505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632499, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:54:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:54:59Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.2.182.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.2.182.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.2.182.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/krb5kdc/rangeradmin_127.2.182.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.2.182.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/krb5kdc/rangeradmin_127.2.182.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.2.182.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.2.182.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.2.182.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/krb5kdc/rangerlookup_127.2.182.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.2.182.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/krb5kdc/rangerlookup_127.2.182.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.182.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.182.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.182.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/krb5kdc/HTTP_127.2.182.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.182.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/krb5kdc/HTTP_127.2.182.84@KRBTEST.COM.keytab.
I20260928 21:54:59.658627 2777 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskZMcOxj/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/postgres -l logfile start
2026-09-28 21:55:02.638 UTC [13535] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 21:55:02.638 UTC [13535] LOG: listening on IPv4 address "127.2.182.84", port 46147
2026-09-28 21:55:02.642 UTC [13535] LOG: listening on Unix socket "/tmp/.s.PGSQL.46147"
2026-09-28 21:55:02.671 UTC [13538] LOG: database system was shut down at 2026-09-28 21:55:00 UTC
2026-09-28 21:55:02.675 UTC [13535] LOG: database system is ready to accept connections
I20260928 21:55:03.547878 2777 mini_postgres.cc:96] Postgres bound to 46147
2026-09-28 21:55:03.553 UTC [13546] FATAL: database "slave" does not exist
127.2.182.84:46147 - accepting connections
I20260928 21:55:03.554240 2777 mini_ranger.cc:161] Starting Ranger...
I20260928 21:55:03.569011 2777 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 21:55:04.195132 2777 mini_ranger.cc:87] Created ranger Postgres database
I20260928 21:55:04.195215 2777 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/ranger-admin
2026-09-28 21:55:04,478 [I] DB FLAVOR :POSTGRES
2026-09-28 21:55:04,479 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 21:55:04,479 [I] Checking connection..
2026-09-28 21:55:04,479 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZMcOxj/build/release/bin/postgresql.jar:/tmp/dist-test-taskZMcOxj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.84:46147/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 21:55:04,727 [I] Checking connection passed.
2026-09-28 21:55:04,727 [I] --------- Verifying version history table ---------
2026-09-28 21:55:04,727 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZMcOxj/build/release/bin/postgresql.jar:/tmp/dist-test-taskZMcOxj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.84:46147/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 21:55:04,972 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 21:55:04,972 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZMcOxj/build/release/bin/postgresql.jar:/tmp/dist-test-taskZMcOxj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.84:46147/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 21:55:05,227 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 21:55:05,227 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 21:55:05,227 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZMcOxj/build/release/bin/postgresql.jar:/tmp/dist-test-taskZMcOxj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.84:46147/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskZMcOxj/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 21:55:05.480 UTC [13610] WARNING: there is no transaction in progress
2026-09-28 21:55:05,486 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 21:55:05,486 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZMcOxj/build/release/bin/postgresql.jar:/tmp/dist-test-taskZMcOxj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.84:46147/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 21:55:05,752 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 21:55:05,753 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 21:55:05,753 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZMcOxj/build/release/bin/postgresql.jar:/tmp/dist-test-taskZMcOxj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.84:46147/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 21:55:06,004 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZMcOxj/build/release/bin/postgresql.jar:/tmp/dist-test-taskZMcOxj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.84:46147/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 21:55:06,265 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZMcOxj/build/release/bin/postgresql.jar:/tmp/dist-test-taskZMcOxj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.84:46147/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-k6lz.c.gcp-upstream.internal','N') ;"
2026-09-28 21:55:06,502 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZMcOxj/build/release/bin/postgresql.jar:/tmp/dist-test-taskZMcOxj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.84:46147/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 21:55:06,739 [I] Table x_portal_user does not exist in database ranger
2026-09-28 21:55:06,739 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZMcOxj/build/release/bin/postgresql.jar:/tmp/dist-test-taskZMcOxj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.84:46147/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 21:55:06,991 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 21:55:06,991 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZMcOxj/build/release/bin/postgresql.jar:/tmp/dist-test-taskZMcOxj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.84:46147/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 21:55:07,240 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZMcOxj/build/release/bin/postgresql.jar:/tmp/dist-test-taskZMcOxj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.84:46147/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 21:55:07,477 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 21:55:07,477 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZMcOxj/build/release/bin/postgresql.jar:/tmp/dist-test-taskZMcOxj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.84:46147/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskZMcOxj/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 21:55:08.561 UTC [13718] WARNING: there is no transaction in progress
2026-09-28 21:55:08.581 UTC [13718] WARNING: there is no transaction in progress
2026-09-28 21:55:08.600 UTC [13718] WARNING: there is no transaction in progress
2026-09-28 21:55:08.618 UTC [13718] WARNING: there is no transaction in progress
2026-09-28 21:55:08.636 UTC [13718] WARNING: there is no transaction in progress
2026-09-28 21:55:08.655 UTC [13718] WARNING: there is no transaction in progress
2026-09-28 21:55:08.719 UTC [13718] WARNING: there is no transaction in progress
2026-09-28 21:55:08.726 UTC [13718] WARNING: there is no transaction in progress
2026-09-28 21:55:08.741 UTC [13718] WARNING: there is no transaction in progress
2026-09-28 21:55:08.754 UTC [13718] WARNING: there is no transaction in progress
2026-09-28 21:55:08.767 UTC [13718] WARNING: there is no transaction in progress
2026-09-28 21:55:08.782 UTC [13718] WARNING: there is no transaction in progress
2026-09-28 21:55:08.791 UTC [13718] WARNING: there is no transaction in progress
2026-09-28 21:55:08.798 UTC [13718] WARNING: there is no transaction in progress
2026-09-28 21:55:08.805 UTC [13718] WARNING: there is no transaction in progress
2026-09-28 21:55:08.812 UTC [13718] WARNING: there is no transaction in progress
2026-09-28 21:55:09,569 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 21:55:09,570 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZMcOxj/build/release/bin/postgresql.jar:/tmp/dist-test-taskZMcOxj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.84:46147/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 21:55:09,822 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 21:55:09,822 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZMcOxj/build/release/bin/postgresql.jar:/tmp/dist-test-taskZMcOxj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.84:46147/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 21:55:10,078 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 21:55:10,078 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZMcOxj/build/release/bin/postgresql.jar:/tmp/dist-test-taskZMcOxj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.84:46147/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 21:55:10,335 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 21:55:10,335 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZMcOxj/build/release/bin/postgresql.jar:/tmp/dist-test-taskZMcOxj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.84:46147/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 21:55:10,583 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZMcOxj/build/release/bin/postgresql.jar:/tmp/dist-test-taskZMcOxj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.84:46147/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 21:55:10,839 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZMcOxj/build/release/bin/postgresql.jar:/tmp/dist-test-taskZMcOxj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.84:46147/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-k6lz.c.gcp-upstream.internal';"
2026-09-28 21:55:11,116 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 21:55:11,116 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZMcOxj/build/release/bin/postgresql.jar:/tmp/dist-test-taskZMcOxj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.84:46147/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 21:55:11,353 [I] DB_PATCHES have already been applied
I20260928 21:55:11.358847 2777 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/ranger-admin:/tmp/dist-test-taskZMcOxj/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskZMcOxj/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskZMcOxj/build/release/bin/postgresql.jar:/tmp/dist-test-taskZMcOxj/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 21:55:11.358915 2777 mini_ranger.cc:193] Using host: 127.2.182.84
I20260928 21:55:11.361971 2777 mini_ranger.cc:227] Ranger admin URL: http://127.2.182.84:34709
Sep 28, 2026 9:55:11 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 9:55:11 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 9:55:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 9:55:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 9:55:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 9:55:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 9:55:12 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 21:55:12.390875 2777 mini_ranger.cc:160] Time spent starting Ranger: real 8.837s user 0.000s sys 0.009s
Sep 28, 2026 9:55:34 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 9:55:34 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 9:55:34 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 9:55:34 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 9:55:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 9:55:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 9:55:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 9:55:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 9:55:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 9:55:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 9:55:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 9:55:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 9:55:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 9:55:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 9:55:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 9:55:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 9:55:34 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 9:55:35 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 21:55:35.645452 2777 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.2.182.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.2.182.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.2.182.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/krb5kdc/rangerkms_127.2.182.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.2.182.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/krb5kdc/rangerkms_127.2.182.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.182.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.2.182.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.182.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/krb5kdc/HTTP_127.2.182.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.182.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/krb5kdc/HTTP_127.2.182.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 21:55:35 dist-test-slave-k6lz krb5kdc[13505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632535, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 21:55:35.744030 2777 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 21:55:35.755375 2777 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 21:55:36.697487 2777 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 21:55:36.697556 2777 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/ranger-kms
I20260928 21:55:36.697573 2777 mini_ranger_kms.cc:227] Using postgres at 127.2.182.84:46147
2026-09-28 21:55:36,865 [I] DB FLAVOR :POSTGRES
2026-09-28 21:55:36,866 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 21:55:36,866 [I] Checking connection
2026-09-28 21:55:36,866 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZMcOxj/build/release/bin/postgresql.jar:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.84:46147/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 21:55:37,120 [I] connection success
2026-09-28 21:55:37,120 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 21:55:37,120 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 21:55:37,120 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZMcOxj/build/release/bin/postgresql.jar:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.84:46147/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 21:55:37,383 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 21:55:37,383 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 21:55:37,383 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 21:55:37,383 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZMcOxj/build/release/bin/postgresql.jar:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.84:46147/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 21:55:37,655 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 21:55:38.222921 2777 mini_ranger_kms.cc:310] Created kms service
I20260928 21:55:38.418054 2777 mini_ranger_kms.cc:326] Created kudu user
I20260928 21:55:38.480204 2777 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 21:55:38 dist-test-slave-k6lz krb5kdc[13505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632535, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.2.182.84@KRBTEST.COM
I20260928 21:55:38.902232 2777 mini_ranger_kms.cc:384] Added ranger policy
I20260928 21:55:38.902822 2777 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/ranger-kms:/tmp/dist-test-taskZMcOxj/build/release/bin/postgresql.jar:/tmp/dist-test-taskZMcOxj/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskZMcOxj/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 21:55:38.902864 2777 mini_ranger_kms.cc:240] Using host: 127.2.182.84
I20260928 21:55:38.906216 2777 mini_ranger_kms.cc:276] Ranger KMS PID: 13899
I20260928 21:55:38.906356 2777 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.2.182.84:56357
Sep 28, 2026 9:55:39 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 9:55:39 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 9:55:40 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 9:55:40 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 9:55:40 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 9:55:40 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 21:55:41.078684 2777 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.335s user 0.000s sys 0.151s
I20260928 21:55:41.078785 2777 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 21:55:41.078864 2777 mini_ranger_kms.cc:401] 127.2.182.84:56357/kms/v1/keys
Sep 28 21:55:41 dist-test-slave-k6lz krb5kdc[13505](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790632541, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.182.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 21:55:41 dist-test-slave-k6lz krb5kdc[13505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632541, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.182.84@KRBTEST.COM for HTTP/127.2.182.84@KRBTEST.COM
Sep 28 21:55:42 dist-test-slave-k6lz krb5kdc[13505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632541, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.182.84@KRBTEST.COM for HTTP/127.2.182.84@KRBTEST.COM
Sep 28 21:55:44 dist-test-slave-k6lz krb5kdc[13505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.2.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab.
Entry for principal kudu/127.2.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab.
I20260928 21:55:44.687779 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:41235
--webserver_interface=127.2.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab
--principal=kudu/127.2.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:45749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.126:41235
--ranger_config_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.182.84:56357/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:55:44.770879 13949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:55:44.771118 13949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:55:44.771149 13949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:55:44.772497 13949 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:55:44.772555 13949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:55:44.772573 13949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:55:44.772589 13949 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:55:44.772603 13949 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:55:44.774257 13949 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:45749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/master-0/wal
--ranger_kms_url=127.2.182.84:56357/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.182.126:41235
--ranger_config_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab
--principal=kudu/127.2.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:41235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13949
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:55:44.774528 13949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:55:44.774735 13949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:55:44.779757 13955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:55:44.779922 13957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:55:44.779826 13949 server_base.cc:1034] running on GCE node
W20260928 21:55:44.779747 13954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:55:44.780294 13949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:55:44.780638 13949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:55:44.781795 13949 hybrid_clock.cc:648] HybridClock initialized: now 1790632544781774 us; error 34 us; skew 500 ppm
Sep 28 21:55:44 dist-test-slave-k6lz krb5kdc[13505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632544, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:55:44.790762 13949 init.cc:377] Logged in from keytab as kudu/127.2.182.126@KRBTEST.COM (short username kudu)
I20260928 21:55:44.791397 13949 webserver.cc:460] Webserver started at http://127.2.182.126:39149/ using document root <none> and password file <none>
I20260928 21:55:44.791692 13949 fs_manager.cc:359] Metadata directory not provided
I20260928 21:55:44.791749 13949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:55:44.791888 13949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 21:55:44 dist-test-slave-k6lz krb5kdc[13505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632544, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.126@KRBTEST.COM for HTTP/127.2.182.84@KRBTEST.COM
I20260928 21:55:44.845678 13949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/master-0/data/instance:
uuid: "32392598c2dc494481ca33a3c0e635a7"
format_stamp: "Formatted at 2026-09-28 21:55:44 on dist-test-slave-k6lz"
server_key: "12f922276fdd08c29a8e5189c8e1bb28"
server_key_iv: "47d875517c58942a37d4c4ed7998896e"
server_key_version: "kuduclusterkey@0"
I20260928 21:55:44.846058 13949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/master-0/wal/instance:
uuid: "32392598c2dc494481ca33a3c0e635a7"
format_stamp: "Formatted at 2026-09-28 21:55:44 on dist-test-slave-k6lz"
server_key: "12f922276fdd08c29a8e5189c8e1bb28"
server_key_iv: "47d875517c58942a37d4c4ed7998896e"
server_key_version: "kuduclusterkey@0"
I20260928 21:55:44.847440 13949 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 21:55:44.859315 13964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:55:44.859543 13949 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 21:55:44.859640 13949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/master-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/master-0/wal
uuid: "32392598c2dc494481ca33a3c0e635a7"
format_stamp: "Formatted at 2026-09-28 21:55:44 on dist-test-slave-k6lz"
server_key: "12f922276fdd08c29a8e5189c8e1bb28"
server_key_iv: "47d875517c58942a37d4c4ed7998896e"
server_key_version: "kuduclusterkey@0"
I20260928 21:55:44.859738 13949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:55:44.875906 13949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:55:44.877910 13949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:55:44.878070 13949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:55:44.883041 13949 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.126:41235
I20260928 21:55:44.883072 14016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.126:41235 every 8 connection(s)
I20260928 21:55:44.883500 13949 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
I20260928 21:55:44.884168 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 13949
I20260928 21:55:44.884254 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/master-0/wal/instance
I20260928 21:55:44.884334 14017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:55:44.884436 2777 external_mini_cluster.cc:1428] Setting key 38d3080d45f722e8b0a47ba3e2cb9102
I20260928 21:55:44.887774 14017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32392598c2dc494481ca33a3c0e635a7: Bootstrap starting.
I20260928 21:55:44.888511 14017 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32392598c2dc494481ca33a3c0e635a7: Neither blocks nor log segments found. Creating new log.
I20260928 21:55:44.888816 14017 log.cc:824] T 00000000000000000000000000000000 P 32392598c2dc494481ca33a3c0e635a7: Log is configured to *not* fsync() on all Append() calls
I20260928 21:55:44.889952 14017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32392598c2dc494481ca33a3c0e635a7: No bootstrap required, opened a new log
Sep 28 21:55:44 dist-test-slave-k6lz krb5kdc[13505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:55:44.893057 14017 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32392598c2dc494481ca33a3c0e635a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32392598c2dc494481ca33a3c0e635a7" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 41235 } }
I20260928 21:55:44.893249 14017 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32392598c2dc494481ca33a3c0e635a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:55:44.893282 14017 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32392598c2dc494481ca33a3c0e635a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32392598c2dc494481ca33a3c0e635a7, State: Initialized, Role: FOLLOWER
I20260928 21:55:44.893481 14017 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32392598c2dc494481ca33a3c0e635a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32392598c2dc494481ca33a3c0e635a7" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 41235 } }
I20260928 21:55:44.893605 14017 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32392598c2dc494481ca33a3c0e635a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:55:44.893651 14017 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32392598c2dc494481ca33a3c0e635a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:55:44.893700 14017 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32392598c2dc494481ca33a3c0e635a7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:55:44.894148 14020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:55:44.885038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47538 (local address 127.2.182.126:41235)
0928 21:55:44.885220 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 21:55:44.885228 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 21:55:44.885251 (+ 23us) server_negotiation.cc:407] Connection header received
0928 21:55:44.885836 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:55:44.885843 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:55:44.885899 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:55:44.886025 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:55:44.886600 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:55:44.887554 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:55:44.888326 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:55:44.888782 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:55:44.888879 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:55:44.891145 (+ 2266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:55:44.891149 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:55:44.891149 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:55:44.891160 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:55:44.892593 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:55:44.892910 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:55:44.892913 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:55:44.892913 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:55:44.892989 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:55:44.893157 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:55:44.893158 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:55:44.893158 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:55:44.893447 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:55:44.893604 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 21:55:44.893802 (+ 198us) server_negotiation.cc:299] Negotiation successful
0928 21:55:44.893935 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":90,"threads_started":1}
I20260928 21:55:44.894573 14017 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32392598c2dc494481ca33a3c0e635a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32392598c2dc494481ca33a3c0e635a7" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 41235 } }
I20260928 21:55:44.894718 14017 leader_election.cc:302] T 00000000000000000000000000000000 P 32392598c2dc494481ca33a3c0e635a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32392598c2dc494481ca33a3c0e635a7; no voters:
I20260928 21:55:44.894913 14017 leader_election.cc:288] T 00000000000000000000000000000000 P 32392598c2dc494481ca33a3c0e635a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:55:44.894964 14022 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32392598c2dc494481ca33a3c0e635a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:55:44.895115 14022 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32392598c2dc494481ca33a3c0e635a7 [term 1 LEADER]: Becoming Leader. State: Replica: 32392598c2dc494481ca33a3c0e635a7, State: Running, Role: LEADER
I20260928 21:55:44.895169 14017 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32392598c2dc494481ca33a3c0e635a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:55:44.895221 14022 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32392598c2dc494481ca33a3c0e635a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32392598c2dc494481ca33a3c0e635a7" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 41235 } }
I20260928 21:55:44.895569 14022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32392598c2dc494481ca33a3c0e635a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32392598c2dc494481ca33a3c0e635a7. Latest consensus state: current_term: 1 leader_uuid: "32392598c2dc494481ca33a3c0e635a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32392598c2dc494481ca33a3c0e635a7" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 41235 } } }
I20260928 21:55:44.895579 14023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32392598c2dc494481ca33a3c0e635a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32392598c2dc494481ca33a3c0e635a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32392598c2dc494481ca33a3c0e635a7" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 41235 } } }
I20260928 21:55:44.895776 14023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32392598c2dc494481ca33a3c0e635a7 [sys.catalog]: This master's current role is: LEADER
I20260928 21:55:44.895944 14022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32392598c2dc494481ca33a3c0e635a7 [sys.catalog]: This master's current role is: LEADER
I20260928 21:55:44.896098 14031 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:55:44.896647 14031 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:55:44.897010 14017 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 21:55:44.904901 14031 catalog_manager.cc:1347] Generated new cluster ID: b9298ce560b9447da82473c4b6f43877
I20260928 21:55:44.904980 14031 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:55:44.921334 14031 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:55:44.922185 14031 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:55:44.931797 14031 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 32392598c2dc494481ca33a3c0e635a7: Generated new TSK 0
I20260928 21:55:44.932128 14031 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.182.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab.
Entry for principal kudu/127.2.182.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab.
I20260928 21:55:45.632138 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.182.84:56357/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.65:0
--local_ip_for_outbound_sockets=127.2.182.65
--webserver_interface=127.2.182.65
--webserver_port=0
--tserver_master_addrs=127.2.182.126:41235
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab
--principal=kudu/127.2.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:45749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:55:45.748001 14062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:55:45.748210 14062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:55:45.748243 14062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:55:45.749557 14062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:55:45.749608 14062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:55:45.749644 14062 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.65
I20260928 21:55:45.751187 14062 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:45749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.2.182.84:56357/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab
--principal=kudu/127.2.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.182.65
--webserver_port=0
--tserver_master_addrs=127.2.182.126:41235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14062
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.65
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:55:45.751468 14062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:55:45.751715 14062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:55:45.757169 14068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:55:45.757261 14067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:55:45.757542 14070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:55:45.758538 14062 server_base.cc:1034] running on GCE node
I20260928 21:55:45.758751 14062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:55:45.759039 14062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:55:45.760223 14062 hybrid_clock.cc:648] HybridClock initialized: now 1790632545760205 us; error 34 us; skew 500 ppm
Sep 28 21:55:45 dist-test-slave-k6lz krb5kdc[13505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632545, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:55:45.762929 14062 init.cc:377] Logged in from keytab as kudu/127.2.182.65@KRBTEST.COM (short username kudu)
I20260928 21:55:45.763449 14062 webserver.cc:460] Webserver started at http://127.2.182.65:45805/ using document root <none> and password file <none>
I20260928 21:55:45.763681 14062 fs_manager.cc:359] Metadata directory not provided
I20260928 21:55:45.763729 14062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:55:45.763859 14062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 21:55:45 dist-test-slave-k6lz krb5kdc[13505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632545, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.65@KRBTEST.COM for HTTP/127.2.182.84@KRBTEST.COM
I20260928 21:55:45.777192 14062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/ts-0/data/instance:
uuid: "d8c6be2fb32648f8a34ef93433967ce9"
format_stamp: "Formatted at 2026-09-28 21:55:45 on dist-test-slave-k6lz"
server_key: "341dda779b9f0553e3cde2e97e6fdcf5"
server_key_iv: "3906548e70da604e58cee8f84220bad8"
server_key_version: "kuduclusterkey@0"
I20260928 21:55:45.777704 14062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/ts-0/wal/instance:
uuid: "d8c6be2fb32648f8a34ef93433967ce9"
format_stamp: "Formatted at 2026-09-28 21:55:45 on dist-test-slave-k6lz"
server_key: "341dda779b9f0553e3cde2e97e6fdcf5"
server_key_iv: "3906548e70da604e58cee8f84220bad8"
server_key_version: "kuduclusterkey@0"
I20260928 21:55:45.780135 14062 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260928 21:55:45.796288 14079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:55:45.796566 14062 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 21:55:45.796674 14062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/ts-0/wal
uuid: "d8c6be2fb32648f8a34ef93433967ce9"
format_stamp: "Formatted at 2026-09-28 21:55:45 on dist-test-slave-k6lz"
server_key: "341dda779b9f0553e3cde2e97e6fdcf5"
server_key_iv: "3906548e70da604e58cee8f84220bad8"
server_key_version: "kuduclusterkey@0"
I20260928 21:55:45.796768 14062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:55:45.807706 14062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:55:45.809599 14062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:55:45.809733 14062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:55:45.809969 14062 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:55:45.810321 14062 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:55:45.810357 14062 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:55:45.810387 14062 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:55:45.810423 14062 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:55:45.817714 14062 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.65:42739
I20260928 21:55:45.818264 14062 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
I20260928 21:55:45.819721 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 14062
I20260928 21:55:45.819816 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/ts-0/wal/instance
I20260928 21:55:45.819996 2777 external_mini_cluster.cc:1428] Setting key 1e37f05db1b52f79c9e7c8c35445f6df
Sep 28 21:55:45 dist-test-slave-k6lz krb5kdc[13505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632545, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.65@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:55:45.819298 14192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.65:42739 every 8 connection(s)
I20260928 21:55:45.832937 14195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:55:45.819105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:41235 (local address 127.2.182.65:40529)
0928 21:55:45.820467 (+ 1362us) negotiation.cc:107] Waiting for socket to connect
0928 21:55:45.820514 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 21:55:45.822781 (+ 2267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:55:45.823175 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:55:45.823180 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:55:45.823473 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:55:45.823759 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:55:45.823766 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:55:45.824828 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:55:45.824829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:55:45.826138 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:55:45.826141 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:55:45.826701 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:55:45.826701 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:55:45.826854 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:55:45.827548 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:55:45.827558 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:55:45.829200 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:55:45.831256 (+ 2056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:55:45.831264 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:55:45.831267 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:55:45.831601 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:55:45.831805 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:55:45.831805 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:55:45.831806 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:55:45.831940 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:55:45.832218 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:55:45.832224 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:55:45.832402 (+ 178us) client_negotiation.cc:769] Sending connection context
0928 21:55:45.832519 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 21:55:45.832682 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1104,"thread_start_us":97,"threads_started":1}
I20260928 21:55:45.833375 14196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:55:45.819460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.65:40529 (local address 127.2.182.126:41235)
0928 21:55:45.819593 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 21:55:45.819595 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:55:45.822175 (+ 2580us) server_negotiation.cc:407] Connection header received
0928 21:55:45.822965 (+ 790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:55:45.822966 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:55:45.822980 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:55:45.823039 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:55:45.823873 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:55:45.824714 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:55:45.826238 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:55:45.826567 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:55:45.826625 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:55:45.829313 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:55:45.829314 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:55:45.829315 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:55:45.829323 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:55:45.831166 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:55:45.831682 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:55:45.831683 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:55:45.831684 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:55:45.831747 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:55:45.832000 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:55:45.832001 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:55:45.832001 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:55:45.832140 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:55:45.832193 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 21:55:45.833195 (+ 1002us) server_negotiation.cc:299] Negotiation successful
0928 21:55:45.833284 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":76,"threads_started":1}
I20260928 21:55:45.833782 14193 heartbeater.cc:347] Connected to a master server at 127.2.182.126:41235
I20260928 21:55:45.833875 14193 heartbeater.cc:464] Registering TS with master...
I20260928 21:55:45.834082 14193 heartbeater.cc:511] Master 127.2.182.126:41235 requested a full tablet report, sending...
I20260928 21:55:45.834664 13981 ts_manager.cc:194] Registered new tserver with Master: d8c6be2fb32648f8a34ef93433967ce9 (127.2.182.65:42739)
I20260928 21:55:45.835695 13981 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.65@KRBTEST.COM'} at 127.2.182.65:40529
WARNING: no policy specified for kudu/127.2.182.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab.
Entry for principal kudu/127.2.182.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab.
I20260928 21:55:45.859020 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.182.84:56357/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.66:0
--local_ip_for_outbound_sockets=127.2.182.66
--webserver_interface=127.2.182.66
--webserver_port=0
--tserver_master_addrs=127.2.182.126:41235
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab
--principal=kudu/127.2.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:45749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 21:55:45 dist-test-slave-k6lz krb5kdc[13505](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790632545, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 21:55:45.998922 14199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:55:45.999137 14199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:55:45.999172 14199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:55:46.000522 14199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:55:46.000569 14199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:55:46.000607 14199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.66
I20260928 21:55:46.002832 14199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:45749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.2.182.84:56357/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab
--principal=kudu/127.2.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.182.66
--webserver_port=0
--tserver_master_addrs=127.2.182.126:41235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.66
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:55:46.003351 14199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:55:46.003695 14199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:55:46.010605 14209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:55:46.010946 14207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:55:46.011205 14206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:55:46.024945 14199 server_base.cc:1034] running on GCE node
I20260928 21:55:46.025245 14199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:55:46.025527 14199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:55:46.032922 14199 hybrid_clock.cc:648] HybridClock initialized: now 1790632546032889 us; error 54 us; skew 500 ppm
Sep 28 21:55:46 dist-test-slave-k6lz krb5kdc[13505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632546, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:55:46.036957 14199 init.cc:377] Logged in from keytab as kudu/127.2.182.66@KRBTEST.COM (short username kudu)
I20260928 21:55:46.037662 14199 webserver.cc:460] Webserver started at http://127.2.182.66:39137/ using document root <none> and password file <none>
I20260928 21:55:46.037923 14199 fs_manager.cc:359] Metadata directory not provided
I20260928 21:55:46.037986 14199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:55:46.038120 14199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 21:55:46 dist-test-slave-k6lz krb5kdc[13505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632546, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.66@KRBTEST.COM for HTTP/127.2.182.84@KRBTEST.COM
I20260928 21:55:46.051290 14199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/ts-1/data/instance:
uuid: "56ff016d14d943f483f722b8e0eaae3f"
format_stamp: "Formatted at 2026-09-28 21:55:46 on dist-test-slave-k6lz"
server_key: "d28502c8eea392f11732f51da74f64fe"
server_key_iv: "1b9539aab673495857d8111191a289ec"
server_key_version: "kuduclusterkey@0"
I20260928 21:55:46.051790 14199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/ts-1/wal/instance:
uuid: "56ff016d14d943f483f722b8e0eaae3f"
format_stamp: "Formatted at 2026-09-28 21:55:46 on dist-test-slave-k6lz"
server_key: "d28502c8eea392f11732f51da74f64fe"
server_key_iv: "1b9539aab673495857d8111191a289ec"
server_key_version: "kuduclusterkey@0"
I20260928 21:55:46.053388 14199 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:55:46.065852 14216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:55:46.066061 14199 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 21:55:46.066159 14199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/ts-1/wal
uuid: "56ff016d14d943f483f722b8e0eaae3f"
format_stamp: "Formatted at 2026-09-28 21:55:46 on dist-test-slave-k6lz"
server_key: "d28502c8eea392f11732f51da74f64fe"
server_key_iv: "1b9539aab673495857d8111191a289ec"
server_key_version: "kuduclusterkey@0"
I20260928 21:55:46.066258 14199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:55:46.075475 14199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:55:46.077836 14199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:55:46.077988 14199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:55:46.078244 14199 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:55:46.078630 14199 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:55:46.078681 14199 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:55:46.078711 14199 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:55:46.078733 14199 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:55:46.085193 14199 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.66:35429
I20260928 21:55:46.085443 14329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.66:35429 every 8 connection(s)
I20260928 21:55:46.085628 14199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
I20260928 21:55:46.089840 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 14199
I20260928 21:55:46.089946 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/ts-1/wal/instance
I20260928 21:55:46.090149 2777 external_mini_cluster.cc:1428] Setting key f8af28e2c489b8db3d18df378d654ed4
Sep 28 21:55:46 dist-test-slave-k6lz krb5kdc[13505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632546, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.66@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:55:46.097530 14196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:55:46.086565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.66:52865 (local address 127.2.182.126:41235)
0928 21:55:46.086609 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 21:55:46.086611 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:55:46.086684 (+ 73us) server_negotiation.cc:407] Connection header received
0928 21:55:46.087345 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:55:46.087347 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:55:46.087364 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:55:46.087395 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:55:46.088200 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:55:46.088928 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:55:46.091275 (+ 2347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:55:46.091584 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:55:46.091650 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:55:46.094293 (+ 2643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:55:46.094295 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:55:46.094296 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:55:46.094304 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:55:46.095795 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:55:46.096423 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:55:46.096425 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:55:46.096426 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:55:46.096491 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:55:46.096801 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:55:46.096802 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:55:46.096802 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:55:46.096946 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:55:46.097000 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 21:55:46.097304 (+ 304us) server_negotiation.cc:299] Negotiation successful
0928 21:55:46.097400 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 21:55:46.097841 14332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:55:46.086311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:41235 (local address 127.2.182.66:52865)
0928 21:55:46.086506 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 21:55:46.086615 (+ 109us) client_negotiation.cc:174] Beginning negotiation
0928 21:55:46.087204 (+ 589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:55:46.087597 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:55:46.087601 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:55:46.087846 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:55:46.088103 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:55:46.088108 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:55:46.089045 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:55:46.089046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:55:46.091144 (+ 2098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:55:46.091148 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:55:46.091673 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:55:46.091674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:55:46.091824 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:55:46.092541 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:55:46.092553 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:55:46.094187 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:55:46.095883 (+ 1696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:55:46.095982 (+ 99us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:55:46.095985 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:55:46.096319 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:55:46.096568 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:55:46.096569 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:55:46.096569 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:55:46.096723 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:55:46.097040 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:55:46.097046 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:55:46.097181 (+ 135us) client_negotiation.cc:769] Sending connection context
0928 21:55:46.097267 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 21:55:46.097422 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":66,"threads_started":1}
I20260928 21:55:46.098281 14330 heartbeater.cc:347] Connected to a master server at 127.2.182.126:41235
I20260928 21:55:46.098387 14330 heartbeater.cc:464] Registering TS with master...
I20260928 21:55:46.098593 14330 heartbeater.cc:511] Master 127.2.182.126:41235 requested a full tablet report, sending...
I20260928 21:55:46.098990 13981 ts_manager.cc:194] Registered new tserver with Master: 56ff016d14d943f483f722b8e0eaae3f (127.2.182.66:35429)
I20260928 21:55:46.099552 13981 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.66@KRBTEST.COM'} at 127.2.182.66:52865
WARNING: no policy specified for kudu/127.2.182.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab.
Entry for principal kudu/127.2.182.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab.
I20260928 21:55:46.122915 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.182.84:56357/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.67:0
--local_ip_for_outbound_sockets=127.2.182.67
--webserver_interface=127.2.182.67
--webserver_port=0
--tserver_master_addrs=127.2.182.126:41235
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab
--principal=kudu/127.2.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:45749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:55:46.270828 14336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:55:46.271181 14336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:55:46.271258 14336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:55:46.273051 14336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:55:46.273294 14336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:55:46.273380 14336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.67
I20260928 21:55:46.275897 14336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:45749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.2.182.84:56357/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab
--principal=kudu/127.2.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.182.67
--webserver_port=0
--tserver_master_addrs=127.2.182.126:41235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.67
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:55:46.276397 14336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:55:46.276737 14336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:55:46.283380 14342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:55:46.283655 14344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:55:46.283443 14341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:55:46.285014 14336 server_base.cc:1034] running on GCE node
I20260928 21:55:46.285215 14336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:55:46.285467 14336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:55:46.286617 14336 hybrid_clock.cc:648] HybridClock initialized: now 1790632546286597 us; error 31 us; skew 500 ppm
Sep 28 21:55:46 dist-test-slave-k6lz krb5kdc[13505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632546, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:55:46.289700 14336 init.cc:377] Logged in from keytab as kudu/127.2.182.67@KRBTEST.COM (short username kudu)
I20260928 21:55:46.290169 14336 webserver.cc:460] Webserver started at http://127.2.182.67:39883/ using document root <none> and password file <none>
I20260928 21:55:46.290414 14336 fs_manager.cc:359] Metadata directory not provided
I20260928 21:55:46.290472 14336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:55:46.290587 14336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 21:55:46 dist-test-slave-k6lz krb5kdc[13505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632546, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.67@KRBTEST.COM for HTTP/127.2.182.84@KRBTEST.COM
I20260928 21:55:46.312788 14336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/ts-2/data/instance:
uuid: "bbee883d516849fa9a9b7fdda6ce7df4"
format_stamp: "Formatted at 2026-09-28 21:55:46 on dist-test-slave-k6lz"
server_key: "ba0241f30420e96824dc2d920e4607db"
server_key_iv: "863f007850148f1579d70c50f5fdb45b"
server_key_version: "kuduclusterkey@0"
I20260928 21:55:46.313298 14336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/ts-2/wal/instance:
uuid: "bbee883d516849fa9a9b7fdda6ce7df4"
format_stamp: "Formatted at 2026-09-28 21:55:46 on dist-test-slave-k6lz"
server_key: "ba0241f30420e96824dc2d920e4607db"
server_key_iv: "863f007850148f1579d70c50f5fdb45b"
server_key_version: "kuduclusterkey@0"
I20260928 21:55:46.314725 14336 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:55:46.327764 14351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:55:46.327946 14336 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:55:46.328040 14336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/ts-2/wal
uuid: "bbee883d516849fa9a9b7fdda6ce7df4"
format_stamp: "Formatted at 2026-09-28 21:55:46 on dist-test-slave-k6lz"
server_key: "ba0241f30420e96824dc2d920e4607db"
server_key_iv: "863f007850148f1579d70c50f5fdb45b"
server_key_version: "kuduclusterkey@0"
I20260928 21:55:46.328143 14336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:55:46.338312 14336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:55:46.340183 14336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:55:46.340317 14336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:55:46.340569 14336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:55:46.340919 14336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:55:46.340965 14336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:55:46.341003 14336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:55:46.341024 14336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:55:46.348500 14336 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.67:42025
I20260928 21:55:46.348963 14336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
I20260928 21:55:46.352558 14464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.67:42025 every 8 connection(s)
I20260928 21:55:46.354041 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 14336
I20260928 21:55:46.354137 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/ts-2/wal/instance
I20260928 21:55:46.354346 2777 external_mini_cluster.cc:1428] Setting key 90286bd92e0ac3420ef607b8246c2df1
Sep 28 21:55:46 dist-test-slave-k6lz krb5kdc[13505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632546, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.67@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:55:46.365252 14196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:55:46.353699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.67:36645 (local address 127.2.182.126:41235)
0928 21:55:46.353751 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 21:55:46.353753 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:55:46.354470 (+ 717us) server_negotiation.cc:407] Connection header received
0928 21:55:46.355385 (+ 915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:55:46.355386 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:55:46.355398 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:55:46.355430 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:55:46.356404 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:55:46.357184 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:55:46.358682 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:55:46.358981 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:55:46.359023 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:55:46.361740 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:55:46.361742 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:55:46.361742 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:55:46.361750 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:55:46.363472 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:55:46.364001 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:55:46.364003 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:55:46.364004 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:55:46.364071 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:55:46.364426 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:55:46.364427 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:55:46.364428 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:55:46.364598 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:55:46.364643 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 21:55:46.365059 (+ 416us) server_negotiation.cc:299] Negotiation successful
0928 21:55:46.365148 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 21:55:46.366020 14467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:55:46.353741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:41235 (local address 127.2.182.67:36645)
0928 21:55:46.354237 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0928 21:55:46.354311 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0928 21:55:46.355203 (+ 892us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:55:46.355639 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:55:46.355644 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:55:46.355996 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:55:46.356291 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:55:46.356299 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:55:46.357300 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:55:46.357301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:55:46.358579 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:55:46.358582 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:55:46.359110 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:55:46.359111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:55:46.359248 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:55:46.359855 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:55:46.359866 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:55:46.361614 (+ 1748us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:55:46.363584 (+ 1970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:55:46.363592 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:55:46.363595 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:55:46.363903 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:55:46.364189 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:55:46.364189 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:55:46.364190 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:55:46.364343 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:55:46.364690 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:55:46.364696 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:55:46.364895 (+ 199us) client_negotiation.cc:769] Sending connection context
0928 21:55:46.365593 (+ 698us) client_negotiation.cc:240] Negotiation successful
0928 21:55:46.365772 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":146,"threads_started":1}
I20260928 21:55:46.366664 14465 heartbeater.cc:347] Connected to a master server at 127.2.182.126:41235
I20260928 21:55:46.366758 14465 heartbeater.cc:464] Registering TS with master...
I20260928 21:55:46.366943 14465 heartbeater.cc:511] Master 127.2.182.126:41235 requested a full tablet report, sending...
I20260928 21:55:46.367386 13981 ts_manager.cc:194] Registered new tserver with Master: bbee883d516849fa9a9b7fdda6ce7df4 (127.2.182.67:42025)
I20260928 21:55:46.367882 13981 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.67@KRBTEST.COM'} at 127.2.182.67:36645
I20260928 21:55:46.376668 2777 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:55:46.395370 14196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:55:46.383686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47552 (local address 127.2.182.126:41235)
0928 21:55:46.383734 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 21:55:46.383736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:55:46.387593 (+ 3857us) server_negotiation.cc:407] Connection header received
0928 21:55:46.387706 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:55:46.387707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:55:46.387719 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:55:46.387749 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:55:46.388605 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:55:46.389358 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:55:46.390417 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:55:46.390726 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:55:46.390769 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:55:46.392406 (+ 1637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:55:46.392408 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:55:46.392408 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:55:46.392416 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:55:46.394041 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:55:46.394454 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:55:46.394455 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:55:46.394456 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:55:46.394525 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:55:46.394728 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:55:46.394728 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:55:46.394729 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:55:46.394863 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:55:46.394935 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 21:55:46.395149 (+ 214us) server_negotiation.cc:299] Negotiation successful
0928 21:55:46.395230 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 21:55:46.397660 13981 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47552:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:55:46.400094 13981 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:55:46.415858 14482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:55:46.403826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.67:42025 (local address 127.0.0.1:35834)
0928 21:55:46.405344 (+ 1518us) negotiation.cc:107] Waiting for socket to connect
0928 21:55:46.405392 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 21:55:46.405463 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:55:46.405887 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:55:46.405891 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:55:46.405893 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:55:46.405967 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:55:46.405969 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:55:46.409898 (+ 3929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:55:46.409899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:55:46.411199 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:55:46.411203 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:55:46.415086 (+ 3883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:55:46.415087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:55:46.415196 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:55:46.415212 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 21:55:46.415288 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 21:55:46.415769 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1485,"thread_start_us":80,"threads_started":1}
I20260928 21:55:46.416736 14478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:55:46.404620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.66:35429 (local address 127.0.0.1:52086)
0928 21:55:46.407244 (+ 2624us) negotiation.cc:107] Waiting for socket to connect
0928 21:55:46.407252 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 21:55:46.407290 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:55:46.407621 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:55:46.407622 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:55:46.407624 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:55:46.407672 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:55:46.407673 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:55:46.408897 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:55:46.408898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:55:46.413853 (+ 4955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:55:46.413857 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:55:46.416459 (+ 2602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:55:46.416460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:55:46.416554 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:55:46.416559 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 21:55:46.416611 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 21:55:46.416658 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2612,"thread_start_us":61,"threads_started":1}
I20260928 21:55:46.417136 14483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:55:46.404855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52086 (local address 127.2.182.66:35429)
0928 21:55:46.405626 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0928 21:55:46.405630 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:55:46.407388 (+ 1758us) server_negotiation.cc:407] Connection header received
0928 21:55:46.407434 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:55:46.407437 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:55:46.407453 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:55:46.407536 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:55:46.407741 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:55:46.408800 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:55:46.414032 (+ 5232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:55:46.414846 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:55:46.414924 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:55:46.415040 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 21:55:46.416993 (+ 1953us) server_negotiation.cc:299] Negotiation successful
0928 21:55:46.417051 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":732,"thread_start_us":72,"threads_started":1}
I20260928 21:55:46.417519 14481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:55:46.404332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35834 (local address 127.2.182.67:42025)
0928 21:55:46.405273 (+ 941us) server_negotiation.cc:206] Beginning negotiation
0928 21:55:46.405278 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:55:46.405654 (+ 376us) server_negotiation.cc:407] Connection header received
0928 21:55:46.405701 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:55:46.405704 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:55:46.405721 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:55:46.405790 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:55:46.406045 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:55:46.409803 (+ 3758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:55:46.412259 (+ 2456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:55:46.413119 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:55:46.413193 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:55:46.413282 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 21:55:46.417356 (+ 4074us) server_negotiation.cc:299] Negotiation successful
0928 21:55:46.417416 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":897,"thread_start_us":75,"threads_started":1}
I20260928 21:55:46.418522 14254 tablet_service.cc:1467] Processing CreateTablet for tablet a5894ade4e3541b1ab45d60c0e8cfd6f (DEFAULT_TABLE table=test-table [id=c5310a0480214efa84a25d19eac48a0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:55:46.418947 14254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5894ade4e3541b1ab45d60c0e8cfd6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:55:46.422679 14479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:55:46.404062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.65:42739 (local address 127.0.0.1:53276)
0928 21:55:46.407117 (+ 3055us) negotiation.cc:107] Waiting for socket to connect
0928 21:55:46.407128 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:55:46.407173 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:55:46.418534 (+ 11361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:55:46.418536 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:55:46.418539 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:55:46.418589 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:55:46.418591 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:55:46.420385 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:55:46.420386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:55:46.421483 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:55:46.421487 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:55:46.422421 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:55:46.422422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:55:46.422495 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:55:46.422514 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 21:55:46.422547 (+ 33us) client_negotiation.cc:240] Negotiation successful
0928 21:55:46.422593 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3035,"thread_start_us":63,"threads_started":1}
I20260928 21:55:46.423290 14480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:55:46.404169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53276 (local address 127.2.182.65:42739)
0928 21:55:46.418339 (+ 14170us) server_negotiation.cc:206] Beginning negotiation
0928 21:55:46.418344 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:55:46.418354 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:55:46.418389 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:55:46.418391 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:55:46.418410 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:55:46.418664 (+ 254us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:55:46.418795 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:55:46.420245 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:55:46.421605 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:55:46.422321 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:55:46.423064 (+ 743us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:55:46.423142 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 21:55:46.423169 (+ 27us) server_negotiation.cc:299] Negotiation successful
0928 21:55:46.423216 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14129,"thread_start_us":66,"threads_started":1}
I20260928 21:55:46.423547 14399 tablet_service.cc:1467] Processing CreateTablet for tablet a5894ade4e3541b1ab45d60c0e8cfd6f (DEFAULT_TABLE table=test-table [id=c5310a0480214efa84a25d19eac48a0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:55:46.423905 14399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5894ade4e3541b1ab45d60c0e8cfd6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:55:46.424094 14127 tablet_service.cc:1467] Processing CreateTablet for tablet a5894ade4e3541b1ab45d60c0e8cfd6f (DEFAULT_TABLE table=test-table [id=c5310a0480214efa84a25d19eac48a0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:55:46.424425 14127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5894ade4e3541b1ab45d60c0e8cfd6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:55:46.431630 14484 tablet_bootstrap.cc:492] T a5894ade4e3541b1ab45d60c0e8cfd6f P 56ff016d14d943f483f722b8e0eaae3f: Bootstrap starting.
I20260928 21:55:46.432870 14484 tablet_bootstrap.cc:654] T a5894ade4e3541b1ab45d60c0e8cfd6f P 56ff016d14d943f483f722b8e0eaae3f: Neither blocks nor log segments found. Creating new log.
I20260928 21:55:46.433228 14484 log.cc:824] T a5894ade4e3541b1ab45d60c0e8cfd6f P 56ff016d14d943f483f722b8e0eaae3f: Log is configured to *not* fsync() on all Append() calls
I20260928 21:55:46.434535 14485 tablet_bootstrap.cc:492] T a5894ade4e3541b1ab45d60c0e8cfd6f P d8c6be2fb32648f8a34ef93433967ce9: Bootstrap starting.
I20260928 21:55:46.436040 14485 tablet_bootstrap.cc:654] T a5894ade4e3541b1ab45d60c0e8cfd6f P d8c6be2fb32648f8a34ef93433967ce9: Neither blocks nor log segments found. Creating new log.
I20260928 21:55:46.436479 14485 log.cc:824] T a5894ade4e3541b1ab45d60c0e8cfd6f P d8c6be2fb32648f8a34ef93433967ce9: Log is configured to *not* fsync() on all Append() calls
I20260928 21:55:46.437002 14486 tablet_bootstrap.cc:492] T a5894ade4e3541b1ab45d60c0e8cfd6f P bbee883d516849fa9a9b7fdda6ce7df4: Bootstrap starting.
I20260928 21:55:46.438074 14486 tablet_bootstrap.cc:654] T a5894ade4e3541b1ab45d60c0e8cfd6f P bbee883d516849fa9a9b7fdda6ce7df4: Neither blocks nor log segments found. Creating new log.
I20260928 21:55:46.438448 14486 log.cc:824] T a5894ade4e3541b1ab45d60c0e8cfd6f P bbee883d516849fa9a9b7fdda6ce7df4: Log is configured to *not* fsync() on all Append() calls
I20260928 21:55:46.440171 14485 tablet_bootstrap.cc:492] T a5894ade4e3541b1ab45d60c0e8cfd6f P d8c6be2fb32648f8a34ef93433967ce9: No bootstrap required, opened a new log
I20260928 21:55:46.440251 14485 ts_tablet_manager.cc:1397] T a5894ade4e3541b1ab45d60c0e8cfd6f P d8c6be2fb32648f8a34ef93433967ce9: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20260928 21:55:46.441236 14484 tablet_bootstrap.cc:492] T a5894ade4e3541b1ab45d60c0e8cfd6f P 56ff016d14d943f483f722b8e0eaae3f: No bootstrap required, opened a new log
I20260928 21:55:46.441300 14484 ts_tablet_manager.cc:1397] T a5894ade4e3541b1ab45d60c0e8cfd6f P 56ff016d14d943f483f722b8e0eaae3f: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260928 21:55:46.442853 14485 raft_consensus.cc:348] T a5894ade4e3541b1ab45d60c0e8cfd6f P d8c6be2fb32648f8a34ef93433967ce9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbee883d516849fa9a9b7fdda6ce7df4" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 42025 } } peers { permanent_uuid: "d8c6be2fb32648f8a34ef93433967ce9" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 42739 } } peers { permanent_uuid: "56ff016d14d943f483f722b8e0eaae3f" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 35429 } }
I20260928 21:55:46.443034 14485 raft_consensus.cc:374] T a5894ade4e3541b1ab45d60c0e8cfd6f P d8c6be2fb32648f8a34ef93433967ce9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:55:46.443065 14485 raft_consensus.cc:729] T a5894ade4e3541b1ab45d60c0e8cfd6f P d8c6be2fb32648f8a34ef93433967ce9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8c6be2fb32648f8a34ef93433967ce9, State: Initialized, Role: FOLLOWER
I20260928 21:55:46.443231 14484 raft_consensus.cc:348] T a5894ade4e3541b1ab45d60c0e8cfd6f P 56ff016d14d943f483f722b8e0eaae3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbee883d516849fa9a9b7fdda6ce7df4" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 42025 } } peers { permanent_uuid: "d8c6be2fb32648f8a34ef93433967ce9" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 42739 } } peers { permanent_uuid: "56ff016d14d943f483f722b8e0eaae3f" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 35429 } }
I20260928 21:55:46.443415 14484 raft_consensus.cc:374] T a5894ade4e3541b1ab45d60c0e8cfd6f P 56ff016d14d943f483f722b8e0eaae3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:55:46.443454 14484 raft_consensus.cc:729] T a5894ade4e3541b1ab45d60c0e8cfd6f P 56ff016d14d943f483f722b8e0eaae3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56ff016d14d943f483f722b8e0eaae3f, State: Initialized, Role: FOLLOWER
I20260928 21:55:46.443581 14484 consensus_queue.cc:260] T a5894ade4e3541b1ab45d60c0e8cfd6f P 56ff016d14d943f483f722b8e0eaae3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbee883d516849fa9a9b7fdda6ce7df4" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 42025 } } peers { permanent_uuid: "d8c6be2fb32648f8a34ef93433967ce9" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 42739 } } peers { permanent_uuid: "56ff016d14d943f483f722b8e0eaae3f" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 35429 } }
I20260928 21:55:46.443572 14485 consensus_queue.cc:260] T a5894ade4e3541b1ab45d60c0e8cfd6f P d8c6be2fb32648f8a34ef93433967ce9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbee883d516849fa9a9b7fdda6ce7df4" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 42025 } } peers { permanent_uuid: "d8c6be2fb32648f8a34ef93433967ce9" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 42739 } } peers { permanent_uuid: "56ff016d14d943f483f722b8e0eaae3f" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 35429 } }
I20260928 21:55:46.443885 14484 ts_tablet_manager.cc:1428] T a5894ade4e3541b1ab45d60c0e8cfd6f P 56ff016d14d943f483f722b8e0eaae3f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 21:55:46.443902 14485 ts_tablet_manager.cc:1428] T a5894ade4e3541b1ab45d60c0e8cfd6f P d8c6be2fb32648f8a34ef93433967ce9: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260928 21:55:46.444027 14330 heartbeater.cc:503] Master 127.2.182.126:41235 was elected leader, sending a full tablet report...
I20260928 21:55:46.444080 14193 heartbeater.cc:503] Master 127.2.182.126:41235 was elected leader, sending a full tablet report...
I20260928 21:55:46.445015 14486 tablet_bootstrap.cc:492] T a5894ade4e3541b1ab45d60c0e8cfd6f P bbee883d516849fa9a9b7fdda6ce7df4: No bootstrap required, opened a new log
I20260928 21:55:46.445096 14486 ts_tablet_manager.cc:1397] T a5894ade4e3541b1ab45d60c0e8cfd6f P bbee883d516849fa9a9b7fdda6ce7df4: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260928 21:55:46.446967 14486 raft_consensus.cc:348] T a5894ade4e3541b1ab45d60c0e8cfd6f P bbee883d516849fa9a9b7fdda6ce7df4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbee883d516849fa9a9b7fdda6ce7df4" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 42025 } } peers { permanent_uuid: "d8c6be2fb32648f8a34ef93433967ce9" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 42739 } } peers { permanent_uuid: "56ff016d14d943f483f722b8e0eaae3f" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 35429 } }
I20260928 21:55:46.447160 14486 raft_consensus.cc:374] T a5894ade4e3541b1ab45d60c0e8cfd6f P bbee883d516849fa9a9b7fdda6ce7df4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:55:46.447198 14486 raft_consensus.cc:729] T a5894ade4e3541b1ab45d60c0e8cfd6f P bbee883d516849fa9a9b7fdda6ce7df4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbee883d516849fa9a9b7fdda6ce7df4, State: Initialized, Role: FOLLOWER
I20260928 21:55:46.447306 14486 consensus_queue.cc:260] T a5894ade4e3541b1ab45d60c0e8cfd6f P bbee883d516849fa9a9b7fdda6ce7df4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbee883d516849fa9a9b7fdda6ce7df4" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 42025 } } peers { permanent_uuid: "d8c6be2fb32648f8a34ef93433967ce9" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 42739 } } peers { permanent_uuid: "56ff016d14d943f483f722b8e0eaae3f" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 35429 } }
I20260928 21:55:46.447623 14486 ts_tablet_manager.cc:1428] T a5894ade4e3541b1ab45d60c0e8cfd6f P bbee883d516849fa9a9b7fdda6ce7df4: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 21:55:46.447762 14465 heartbeater.cc:503] Master 127.2.182.126:41235 was elected leader, sending a full tablet report...
I20260928 21:55:46.478041 14492 raft_consensus.cc:482] T a5894ade4e3541b1ab45d60c0e8cfd6f P bbee883d516849fa9a9b7fdda6ce7df4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:55:46.478199 14492 raft_consensus.cc:504] T a5894ade4e3541b1ab45d60c0e8cfd6f P bbee883d516849fa9a9b7fdda6ce7df4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbee883d516849fa9a9b7fdda6ce7df4" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 42025 } } peers { permanent_uuid: "d8c6be2fb32648f8a34ef93433967ce9" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 42739 } } peers { permanent_uuid: "56ff016d14d943f483f722b8e0eaae3f" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 35429 } }
I20260928 21:55:46.482440 14467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:55:46.478744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.65:42739 (local address 127.2.182.67:57127)
0928 21:55:46.478850 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0928 21:55:46.478862 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:55:46.478898 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:55:46.479134 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:55:46.479136 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:55:46.479137 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:55:46.479203 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:55:46.479205 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:55:46.480274 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:55:46.480276 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:55:46.481410 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:55:46.481413 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:55:46.482189 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:55:46.482190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:55:46.482273 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:55:46.482277 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:55:46.482317 (+ 40us) client_negotiation.cc:240] Negotiation successful
0928 21:55:46.482358 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88}
I20260928 21:55:46.482755 14480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:55:46.478965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.67:57127 (local address 127.2.182.65:42739)
0928 21:55:46.479007 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 21:55:46.479008 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:55:46.479016 (+ 8us) server_negotiation.cc:407] Connection header received
0928 21:55:46.479041 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:55:46.479041 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:55:46.479051 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:55:46.479089 (+ 38us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:55:46.479259 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:55:46.480172 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:55:46.481526 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:55:46.482075 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:55:46.482144 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:55:46.482155 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 21:55:46.482647 (+ 492us) server_negotiation.cc:299] Negotiation successful
0928 21:55:46.482695 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 21:55:46.483011 14147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5894ade4e3541b1ab45d60c0e8cfd6f" candidate_uuid: "bbee883d516849fa9a9b7fdda6ce7df4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8c6be2fb32648f8a34ef93433967ce9" is_pre_election: true
I20260928 21:55:46.483177 14147 raft_consensus.cc:2454] T a5894ade4e3541b1ab45d60c0e8cfd6f P d8c6be2fb32648f8a34ef93433967ce9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbee883d516849fa9a9b7fdda6ce7df4 in term 0.
I20260928 21:55:46.483433 14354 leader_election.cc:302] T a5894ade4e3541b1ab45d60c0e8cfd6f P bbee883d516849fa9a9b7fdda6ce7df4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bbee883d516849fa9a9b7fdda6ce7df4, d8c6be2fb32648f8a34ef93433967ce9; no voters:
I20260928 21:55:46.487140 14467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:55:46.483825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.66:35429 (local address 127.2.182.67:35351)
0928 21:55:46.483975 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 21:55:46.483984 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 21:55:46.484020 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:55:46.484167 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:55:46.484168 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:55:46.484169 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:55:46.484205 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:55:46.484206 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:55:46.485196 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:55:46.485197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:55:46.486163 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:55:46.486167 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:55:46.486907 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:55:46.486908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:55:46.486968 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:55:46.486972 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:55:46.487017 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 21:55:46.487059 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137}
I20260928 21:55:46.487457 14483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:55:46.483911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.67:35351 (local address 127.2.182.66:35429)
0928 21:55:46.483951 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 21:55:46.483953 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:55:46.484061 (+ 108us) server_negotiation.cc:407] Connection header received
0928 21:55:46.484085 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:55:46.484086 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:55:46.484096 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:55:46.484131 (+ 35us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:55:46.484257 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:55:46.485095 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:55:46.486262 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:55:46.486785 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:55:46.486859 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:55:46.486871 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 21:55:46.487323 (+ 452us) server_negotiation.cc:299] Negotiation successful
0928 21:55:46.487392 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 21:55:46.487669 14492 leader_election.cc:288] T a5894ade4e3541b1ab45d60c0e8cfd6f P bbee883d516849fa9a9b7fdda6ce7df4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8c6be2fb32648f8a34ef93433967ce9 (127.2.182.65:42739), 56ff016d14d943f483f722b8e0eaae3f (127.2.182.66:35429)
I20260928 21:55:46.487738 14492 raft_consensus.cc:2781] T a5894ade4e3541b1ab45d60c0e8cfd6f P bbee883d516849fa9a9b7fdda6ce7df4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:55:46.487805 14492 raft_consensus.cc:482] T a5894ade4e3541b1ab45d60c0e8cfd6f P bbee883d516849fa9a9b7fdda6ce7df4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:55:46.487833 14492 raft_consensus.cc:3037] T a5894ade4e3541b1ab45d60c0e8cfd6f P bbee883d516849fa9a9b7fdda6ce7df4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:55:46.487784 14282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5894ade4e3541b1ab45d60c0e8cfd6f" candidate_uuid: "bbee883d516849fa9a9b7fdda6ce7df4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56ff016d14d943f483f722b8e0eaae3f" is_pre_election: true
I20260928 21:55:46.487974 14282 raft_consensus.cc:2454] T a5894ade4e3541b1ab45d60c0e8cfd6f P 56ff016d14d943f483f722b8e0eaae3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbee883d516849fa9a9b7fdda6ce7df4 in term 0.
I20260928 21:55:46.488761 14492 raft_consensus.cc:504] T a5894ade4e3541b1ab45d60c0e8cfd6f P bbee883d516849fa9a9b7fdda6ce7df4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbee883d516849fa9a9b7fdda6ce7df4" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 42025 } } peers { permanent_uuid: "d8c6be2fb32648f8a34ef93433967ce9" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 42739 } } peers { permanent_uuid: "56ff016d14d943f483f722b8e0eaae3f" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 35429 } }
I20260928 21:55:46.489156 14147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5894ade4e3541b1ab45d60c0e8cfd6f" candidate_uuid: "bbee883d516849fa9a9b7fdda6ce7df4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8c6be2fb32648f8a34ef93433967ce9"
I20260928 21:55:46.489271 14147 raft_consensus.cc:3037] T a5894ade4e3541b1ab45d60c0e8cfd6f P d8c6be2fb32648f8a34ef93433967ce9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:55:46.490182 14147 raft_consensus.cc:2454] T a5894ade4e3541b1ab45d60c0e8cfd6f P d8c6be2fb32648f8a34ef93433967ce9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbee883d516849fa9a9b7fdda6ce7df4 in term 1.
I20260928 21:55:46.490393 14354 leader_election.cc:302] T a5894ade4e3541b1ab45d60c0e8cfd6f P bbee883d516849fa9a9b7fdda6ce7df4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bbee883d516849fa9a9b7fdda6ce7df4, d8c6be2fb32648f8a34ef93433967ce9; no voters:
I20260928 21:55:46.490636 14282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5894ade4e3541b1ab45d60c0e8cfd6f" candidate_uuid: "bbee883d516849fa9a9b7fdda6ce7df4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56ff016d14d943f483f722b8e0eaae3f"
I20260928 21:55:46.490731 14282 raft_consensus.cc:3037] T a5894ade4e3541b1ab45d60c0e8cfd6f P 56ff016d14d943f483f722b8e0eaae3f [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:55:46.491645 14282 raft_consensus.cc:2454] T a5894ade4e3541b1ab45d60c0e8cfd6f P 56ff016d14d943f483f722b8e0eaae3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbee883d516849fa9a9b7fdda6ce7df4 in term 1.
I20260928 21:55:46.491851 14492 leader_election.cc:288] T a5894ade4e3541b1ab45d60c0e8cfd6f P bbee883d516849fa9a9b7fdda6ce7df4 [CANDIDATE]: Term 1 election: Requested vote from peers d8c6be2fb32648f8a34ef93433967ce9 (127.2.182.65:42739), 56ff016d14d943f483f722b8e0eaae3f (127.2.182.66:35429)
I20260928 21:55:46.491926 14492 raft_consensus.cc:2781] T a5894ade4e3541b1ab45d60c0e8cfd6f P bbee883d516849fa9a9b7fdda6ce7df4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:55:46.491968 14492 raft_consensus.cc:686] T a5894ade4e3541b1ab45d60c0e8cfd6f P bbee883d516849fa9a9b7fdda6ce7df4 [term 1 LEADER]: Becoming Leader. State: Replica: bbee883d516849fa9a9b7fdda6ce7df4, State: Running, Role: LEADER
I20260928 21:55:46.492059 14492 consensus_queue.cc:237] T a5894ade4e3541b1ab45d60c0e8cfd6f P bbee883d516849fa9a9b7fdda6ce7df4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbee883d516849fa9a9b7fdda6ce7df4" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 42025 } } peers { permanent_uuid: "d8c6be2fb32648f8a34ef93433967ce9" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 42739 } } peers { permanent_uuid: "56ff016d14d943f483f722b8e0eaae3f" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 35429 } }
I20260928 21:55:46.492911 13981 catalog_manager.cc:5495] T a5894ade4e3541b1ab45d60c0e8cfd6f P bbee883d516849fa9a9b7fdda6ce7df4 reported cstate change: term changed from 0 to 1, leader changed from <none> to bbee883d516849fa9a9b7fdda6ce7df4 (127.2.182.67). New cstate: current_term: 1 leader_uuid: "bbee883d516849fa9a9b7fdda6ce7df4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbee883d516849fa9a9b7fdda6ce7df4" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 42025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8c6be2fb32648f8a34ef93433967ce9" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 42739 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56ff016d14d943f483f722b8e0eaae3f" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 35429 } health_report { overall_health: UNKNOWN } } }
I20260928 21:55:46.501742 14481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:55:46.498420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35840 (local address 127.2.182.67:42025)
0928 21:55:46.498472 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 21:55:46.498474 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:55:46.498485 (+ 11us) server_negotiation.cc:407] Connection header received
0928 21:55:46.498548 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:55:46.498549 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:55:46.498567 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:55:46.498633 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:55:46.498786 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:55:46.499540 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:55:46.500593 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:55:46.500894 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:55:46.500945 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:55:46.501188 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:55:46.501316 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:55:46.501383 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 21:55:46.501615 (+ 232us) server_negotiation.cc:299] Negotiation successful
0928 21:55:46.501649 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 21:55:46.503762 14282 raft_consensus.cc:1261] T a5894ade4e3541b1ab45d60c0e8cfd6f P 56ff016d14d943f483f722b8e0eaae3f [term 1 FOLLOWER]: Refusing update from remote peer bbee883d516849fa9a9b7fdda6ce7df4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:55:46.504084 14493 consensus_queue.cc:1035] T a5894ade4e3541b1ab45d60c0e8cfd6f P bbee883d516849fa9a9b7fdda6ce7df4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56ff016d14d943f483f722b8e0eaae3f" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 35429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:55:46.506106 14147 raft_consensus.cc:1261] T a5894ade4e3541b1ab45d60c0e8cfd6f P d8c6be2fb32648f8a34ef93433967ce9 [term 1 FOLLOWER]: Refusing update from remote peer bbee883d516849fa9a9b7fdda6ce7df4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:55:46.506353 14493 consensus_queue.cc:1035] T a5894ade4e3541b1ab45d60c0e8cfd6f P bbee883d516849fa9a9b7fdda6ce7df4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8c6be2fb32648f8a34ef93433967ce9" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 42739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:55:46.509965 14496 mvcc.cc:205] Tried to move back new op lower bound from 7334430910476419072 to 7334430910432198656. Current Snapshot: MvccSnapshot[applied={T|T < 7334430910476419072}]
I20260928 21:55:46.510425 14499 mvcc.cc:205] Tried to move back new op lower bound from 7334430910476419072 to 7334430910432198656. Current Snapshot: MvccSnapshot[applied={T|T < 7334430910476419072}]
I20260928 21:55:46.529603 14497 mvcc.cc:205] Tried to move back new op lower bound from 7334430910476419072 to 7334430910432198656. Current Snapshot: MvccSnapshot[applied={T|T < 7334430910476419072}]
I20260928 21:55:46.534420 13981 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47552:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:55:46.534545 13981 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47552:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:55:46.535444 13981 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 32392598c2dc494481ca33a3c0e635a7: Sending DeleteTablet for 3 replicas of tablet a5894ade4e3541b1ab45d60c0e8cfd6f
I20260928 21:55:46.535825 14127 tablet_service.cc:1514] Processing DeleteTablet for tablet a5894ade4e3541b1ab45d60c0e8cfd6f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:55:46 UTC) from {username='kudu', principal='kudu/127.2.182.126@KRBTEST.COM'} at 127.0.0.1:53276
I20260928 21:55:46.536227 14254 tablet_service.cc:1514] Processing DeleteTablet for tablet a5894ade4e3541b1ab45d60c0e8cfd6f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:55:46 UTC) from {username='kudu', principal='kudu/127.2.182.126@KRBTEST.COM'} at 127.0.0.1:52086
I20260928 21:55:46.536721 14399 tablet_service.cc:1514] Processing DeleteTablet for tablet a5894ade4e3541b1ab45d60c0e8cfd6f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:55:46 UTC) from {username='kudu', principal='kudu/127.2.182.126@KRBTEST.COM'} at 127.0.0.1:35834
I20260928 21:55:46.537009 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 14062
I20260928 21:55:46.546038 14507 tablet_replica.cc:330] T a5894ade4e3541b1ab45d60c0e8cfd6f P bbee883d516849fa9a9b7fdda6ce7df4: stopping tablet replica
I20260928 21:55:46.546207 14507 raft_consensus.cc:2229] T a5894ade4e3541b1ab45d60c0e8cfd6f P bbee883d516849fa9a9b7fdda6ce7df4 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:55:46.546384 14507 raft_consensus.cc:2258] T a5894ade4e3541b1ab45d60c0e8cfd6f P bbee883d516849fa9a9b7fdda6ce7df4 [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 28 21:55:46 dist-test-slave-k6lz krb5kdc[13505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632545, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.126@KRBTEST.COM for HTTP/127.2.182.84@KRBTEST.COM
I20260928 21:55:46.546562 14506 tablet_replica.cc:330] T a5894ade4e3541b1ab45d60c0e8cfd6f P 56ff016d14d943f483f722b8e0eaae3f: stopping tablet replica
I20260928 21:55:46.550196 14506 raft_consensus.cc:2229] T a5894ade4e3541b1ab45d60c0e8cfd6f P 56ff016d14d943f483f722b8e0eaae3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:55:46.550491 14506 raft_consensus.cc:2258] T a5894ade4e3541b1ab45d60c0e8cfd6f P 56ff016d14d943f483f722b8e0eaae3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:55:46.550737 14507 ts_tablet_manager.cc:1905] T a5894ade4e3541b1ab45d60c0e8cfd6f P bbee883d516849fa9a9b7fdda6ce7df4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:55:46.551188 14506 ts_tablet_manager.cc:1905] T a5894ade4e3541b1ab45d60c0e8cfd6f P 56ff016d14d943f483f722b8e0eaae3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:55:46.554319 14506 ts_tablet_manager.cc:1918] T a5894ade4e3541b1ab45d60c0e8cfd6f P 56ff016d14d943f483f722b8e0eaae3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:55:46.556528 14507 ts_tablet_manager.cc:1918] T a5894ade4e3541b1ab45d60c0e8cfd6f P bbee883d516849fa9a9b7fdda6ce7df4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:55:46.556617 14507 log.cc:1196] T a5894ade4e3541b1ab45d60c0e8cfd6f P bbee883d516849fa9a9b7fdda6ce7df4: Deleting WAL directory at /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/ts-2/wal/wals/a5894ade4e3541b1ab45d60c0e8cfd6f
I20260928 21:55:46.556744 14506 log.cc:1196] T a5894ade4e3541b1ab45d60c0e8cfd6f P 56ff016d14d943f483f722b8e0eaae3f: Deleting WAL directory at /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632440055392-2777-0/minicluster-data/ts-1/wal/wals/a5894ade4e3541b1ab45d60c0e8cfd6f
I20260928 21:55:46.556869 14507 ts_tablet_manager.cc:1939] T a5894ade4e3541b1ab45d60c0e8cfd6f P bbee883d516849fa9a9b7fdda6ce7df4: Deleting consensus metadata
I20260928 21:55:46.557298 13967 catalog_manager.cc:4841] TS bbee883d516849fa9a9b7fdda6ce7df4 (127.2.182.67:42025): tablet a5894ade4e3541b1ab45d60c0e8cfd6f (table test-table [id=c5310a0480214efa84a25d19eac48a0d]) successfully deleted
I20260928 21:55:46.557605 14506 ts_tablet_manager.cc:1939] T a5894ade4e3541b1ab45d60c0e8cfd6f P 56ff016d14d943f483f722b8e0eaae3f: Deleting consensus metadata
I20260928 21:55:46.558029 13966 catalog_manager.cc:4841] TS 56ff016d14d943f483f722b8e0eaae3f (127.2.182.66:35429): tablet a5894ade4e3541b1ab45d60c0e8cfd6f (table test-table [id=c5310a0480214efa84a25d19eac48a0d]) successfully deleted
W20260928 21:55:46.558256 13967 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.182.65:42739 (error 108)
I20260928 21:55:46.558904 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 14199
I20260928 21:55:46.583925 14479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:55:46.583760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.65:42739 (local address 127.0.0.1:53290)
0928 21:55:46.583819 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 21:55:46.583881 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.65:42739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 21:55:46.584272 13967 catalog_manager.cc:4584] TS d8c6be2fb32648f8a34ef93433967ce9 (127.2.182.65:42739): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a5894ade4e3541b1ab45d60c0e8cfd6f: Network error: Client connection negotiation failed: client connection to 127.2.182.65:42739: connect: Connection refused (error 111)
I20260928 21:55:46.600489 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 14336
I20260928 21:55:46.607517 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 13949
I20260928 21:55:46.614833 2777 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 21:55:46.960114 2777 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 21:55:46.960246 2777 mini_ranger.cc:66] Stopping Ranger...
I20260928 21:55:47.294967 2777 mini_ranger.cc:68] Stopped Ranger
2026-09-28 21:55:47.295 UTC [13535] LOG: received smart shutdown request
2026-09-28 21:55:47.298 UTC [13535] LOG: background worker "logical replication launcher" (PID 13544) exited with exit code 1
2026-09-28 21:55:47.299 UTC [13539] LOG: shutting down
2026-09-28 21:55:47.506 UTC [13535] LOG: database system is shut down
2026-09-28T21:55:47Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49056 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:55:47 dist-test-slave-k6lz krb5kdc[14517](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:55:47 dist-test-slave-k6lz krb5kdc[14517](info): set up 2 sockets
Sep 28 21:55:47 dist-test-slave-k6lz krb5kdc[14517](info): commencing operation
krb5kdc: starting...
W20260928 21:55:48.555476 2777 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Sep 28 21:55:48 dist-test-slave-k6lz krb5kdc[14517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:55:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:55:48Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.2.182.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.2.182.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.2.182.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/krb5kdc/rangeradmin_127.2.182.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.2.182.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/krb5kdc/rangeradmin_127.2.182.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.2.182.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.2.182.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.2.182.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/krb5kdc/rangerlookup_127.2.182.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.2.182.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/krb5kdc/rangerlookup_127.2.182.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.182.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.182.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.182.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/krb5kdc/HTTP_127.2.182.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.182.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/krb5kdc/HTTP_127.2.182.84@KRBTEST.COM.keytab.
I20260928 21:55:48.730825 2777 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskZMcOxj/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/postgres -l logfile start
2026-09-28 21:55:52.071 UTC [14547] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 21:55:52.071 UTC [14547] LOG: listening on IPv4 address "127.2.182.84", port 57477
2026-09-28 21:55:52.076 UTC [14547] LOG: listening on Unix socket "/tmp/.s.PGSQL.57477"
2026-09-28 21:55:52.104 UTC [14550] LOG: database system was shut down at 2026-09-28 21:55:49 UTC
2026-09-28 21:55:52.108 UTC [14547] LOG: database system is ready to accept connections
I20260928 21:55:52.987061 2777 mini_postgres.cc:96] Postgres bound to 57477
2026-09-28 21:55:52.993 UTC [14558] FATAL: database "slave" does not exist
127.2.182.84:57477 - accepting connections
I20260928 21:55:52.993527 2777 mini_ranger.cc:161] Starting Ranger...
I20260928 21:55:53.009732 2777 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 21:55:53.706962 2777 mini_ranger.cc:87] Created ranger Postgres database
I20260928 21:55:53.707038 2777 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ranger-admin
2026-09-28 21:55:54,002 [I] DB FLAVOR :POSTGRES
2026-09-28 21:55:54,003 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 21:55:54,003 [I] Checking connection..
2026-09-28 21:55:54,003 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZMcOxj/build/release/bin/postgresql.jar:/tmp/dist-test-taskZMcOxj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.84:57477/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 21:55:54,270 [I] Checking connection passed.
2026-09-28 21:55:54,270 [I] --------- Verifying version history table ---------
2026-09-28 21:55:54,270 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZMcOxj/build/release/bin/postgresql.jar:/tmp/dist-test-taskZMcOxj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.84:57477/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 21:55:54,543 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 21:55:54,543 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZMcOxj/build/release/bin/postgresql.jar:/tmp/dist-test-taskZMcOxj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.84:57477/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 21:55:54,797 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 21:55:54,797 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 21:55:54,798 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZMcOxj/build/release/bin/postgresql.jar:/tmp/dist-test-taskZMcOxj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.84:57477/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskZMcOxj/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 21:55:55.028 UTC [14622] WARNING: there is no transaction in progress
2026-09-28 21:55:55,034 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 21:55:55,034 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZMcOxj/build/release/bin/postgresql.jar:/tmp/dist-test-taskZMcOxj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.84:57477/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 21:55:55,299 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 21:55:55,300 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 21:55:55,300 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZMcOxj/build/release/bin/postgresql.jar:/tmp/dist-test-taskZMcOxj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.84:57477/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 21:55:55,553 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZMcOxj/build/release/bin/postgresql.jar:/tmp/dist-test-taskZMcOxj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.84:57477/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 21:55:55,814 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZMcOxj/build/release/bin/postgresql.jar:/tmp/dist-test-taskZMcOxj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.84:57477/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-k6lz.c.gcp-upstream.internal','N') ;"
2026-09-28 21:55:56,055 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZMcOxj/build/release/bin/postgresql.jar:/tmp/dist-test-taskZMcOxj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.84:57477/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 21:55:56,320 [I] Table x_portal_user does not exist in database ranger
2026-09-28 21:55:56,321 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZMcOxj/build/release/bin/postgresql.jar:/tmp/dist-test-taskZMcOxj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.84:57477/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 21:55:56,586 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 21:55:56,586 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZMcOxj/build/release/bin/postgresql.jar:/tmp/dist-test-taskZMcOxj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.84:57477/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 21:55:56,829 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZMcOxj/build/release/bin/postgresql.jar:/tmp/dist-test-taskZMcOxj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.84:57477/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 21:55:57,062 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 21:55:57,062 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZMcOxj/build/release/bin/postgresql.jar:/tmp/dist-test-taskZMcOxj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.84:57477/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskZMcOxj/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 21:55:58.218 UTC [14730] WARNING: there is no transaction in progress
2026-09-28 21:55:58.241 UTC [14730] WARNING: there is no transaction in progress
2026-09-28 21:55:58.263 UTC [14730] WARNING: there is no transaction in progress
2026-09-28 21:55:58.282 UTC [14730] WARNING: there is no transaction in progress
2026-09-28 21:55:58.304 UTC [14730] WARNING: there is no transaction in progress
2026-09-28 21:55:58.326 UTC [14730] WARNING: there is no transaction in progress
2026-09-28 21:55:58.389 UTC [14730] WARNING: there is no transaction in progress
2026-09-28 21:55:58.397 UTC [14730] WARNING: there is no transaction in progress
2026-09-28 21:55:58.413 UTC [14730] WARNING: there is no transaction in progress
2026-09-28 21:55:58.424 UTC [14730] WARNING: there is no transaction in progress
2026-09-28 21:55:58.436 UTC [14730] WARNING: there is no transaction in progress
2026-09-28 21:55:58.452 UTC [14730] WARNING: there is no transaction in progress
2026-09-28 21:55:58.464 UTC [14730] WARNING: there is no transaction in progress
2026-09-28 21:55:58.470 UTC [14730] WARNING: there is no transaction in progress
2026-09-28 21:55:58.478 UTC [14730] WARNING: there is no transaction in progress
2026-09-28 21:55:58.487 UTC [14730] WARNING: there is no transaction in progress
2026-09-28 21:55:59,274 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 21:55:59,274 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZMcOxj/build/release/bin/postgresql.jar:/tmp/dist-test-taskZMcOxj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.84:57477/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 21:55:59,527 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 21:55:59,527 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZMcOxj/build/release/bin/postgresql.jar:/tmp/dist-test-taskZMcOxj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.84:57477/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 21:55:59,792 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 21:55:59,793 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZMcOxj/build/release/bin/postgresql.jar:/tmp/dist-test-taskZMcOxj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.84:57477/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 21:56:00,069 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 21:56:00,070 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZMcOxj/build/release/bin/postgresql.jar:/tmp/dist-test-taskZMcOxj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.84:57477/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 21:56:00,330 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZMcOxj/build/release/bin/postgresql.jar:/tmp/dist-test-taskZMcOxj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.84:57477/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 21:56:00,604 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZMcOxj/build/release/bin/postgresql.jar:/tmp/dist-test-taskZMcOxj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.84:57477/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-k6lz.c.gcp-upstream.internal';"
2026-09-28 21:56:00,877 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 21:56:00,878 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZMcOxj/build/release/bin/postgresql.jar:/tmp/dist-test-taskZMcOxj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.84:57477/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 21:56:01,149 [I] DB_PATCHES have already been applied
I20260928 21:56:01.155771 2777 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ranger-admin:/tmp/dist-test-taskZMcOxj/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskZMcOxj/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskZMcOxj/build/release/bin/postgresql.jar:/tmp/dist-test-taskZMcOxj/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 21:56:01.155839 2777 mini_ranger.cc:193] Using host: 127.2.182.84
I20260928 21:56:01.158948 2777 mini_ranger.cc:227] Ranger admin URL: http://127.2.182.84:58701
Sep 28, 2026 9:56:01 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 9:56:01 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 9:56:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 9:56:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 9:56:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 9:56:02 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 9:56:02 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 21:56:02.202091 2777 mini_ranger.cc:160] Time spent starting Ranger: real 9.209s user 0.000s sys 0.009s
Sep 28, 2026 9:56:24 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 9:56:24 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 9:56:24 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 9:56:24 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 9:56:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 9:56:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 9:56:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 9:56:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 9:56:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 9:56:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 9:56:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 9:56:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 9:56:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 9:56:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 9:56:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 9:56:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 9:56:24 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 9:56:25 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 21:56:25.647722 2777 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.2.182.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.2.182.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.2.182.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/krb5kdc/rangerkms_127.2.182.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.2.182.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/krb5kdc/rangerkms_127.2.182.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.182.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.2.182.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.182.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/krb5kdc/HTTP_127.2.182.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.182.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/krb5kdc/HTTP_127.2.182.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 21:56:25 dist-test-slave-k6lz krb5kdc[14517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632585, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 21:56:25.751816 2777 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 21:56:25.761613 2777 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 21:56:26.717248 2777 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 21:56:26.717330 2777 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ranger-kms
I20260928 21:56:26.717347 2777 mini_ranger_kms.cc:227] Using postgres at 127.2.182.84:57477
2026-09-28 21:56:26,863 [I] DB FLAVOR :POSTGRES
2026-09-28 21:56:26,863 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 21:56:26,864 [I] Checking connection
2026-09-28 21:56:26,864 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZMcOxj/build/release/bin/postgresql.jar:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.84:57477/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 21:56:27,120 [I] connection success
2026-09-28 21:56:27,120 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 21:56:27,120 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 21:56:27,120 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZMcOxj/build/release/bin/postgresql.jar:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.84:57477/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 21:56:27,363 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 21:56:27,363 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 21:56:27,364 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 21:56:27,364 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZMcOxj/build/release/bin/postgresql.jar:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.182.84:57477/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 21:56:27,624 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 21:56:28.132122 2777 mini_ranger_kms.cc:310] Created kms service
I20260928 21:56:28.321659 2777 mini_ranger_kms.cc:326] Created kudu user
I20260928 21:56:28.366331 2777 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 21:56:28 dist-test-slave-k6lz krb5kdc[14517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632585, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.2.182.84@KRBTEST.COM
I20260928 21:56:28.731945 2777 mini_ranger_kms.cc:384] Added ranger policy
I20260928 21:56:28.732059 2777 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ranger-kms:/tmp/dist-test-taskZMcOxj/build/release/bin/postgresql.jar:/tmp/dist-test-taskZMcOxj/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskZMcOxj/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 21:56:28.732075 2777 mini_ranger_kms.cc:240] Using host: 127.2.182.84
I20260928 21:56:28.735303 2777 mini_ranger_kms.cc:276] Ranger KMS PID: 14911
I20260928 21:56:28.735409 2777 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.2.182.84:37893
Sep 28, 2026 9:56:29 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 9:56:29 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 9:56:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 9:56:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 9:56:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 9:56:30 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 28 21:56:30 dist-test-slave-k6lz krb5kdc[14517](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790632590, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.182.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:56:30.835072 2777 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.083s user 0.009s sys 0.121s
I20260928 21:56:30.835175 2777 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 21:56:30.835218 2777 mini_ranger_kms.cc:401] 127.2.182.84:37893/kms/v1/keys
Sep 28 21:56:31 dist-test-slave-k6lz krb5kdc[14517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632590, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.182.84@KRBTEST.COM for HTTP/127.2.182.84@KRBTEST.COM
Sep 28 21:56:31 dist-test-slave-k6lz krb5kdc[14517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632590, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.182.84@KRBTEST.COM for HTTP/127.2.182.84@KRBTEST.COM
Sep 28 21:56:33 dist-test-slave-k6lz krb5kdc[14517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.2.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab.
Entry for principal kudu/127.2.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab.
I20260928 21:56:33.955492 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:40265
--webserver_interface=127.2.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab
--principal=kudu/127.2.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:43655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.126:40265
--ranger_config_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.182.84:37893/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:56:34.038234 14961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:56:34.038434 14961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:56:34.038453 14961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:56:34.039652 14961 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:56:34.039698 14961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:56:34.039713 14961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:56:34.039724 14961 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:56:34.039737 14961 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:56:34.041324 14961 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:43655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/master-0/wal
--ranger_kms_url=127.2.182.84:37893/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.182.126:40265
--ranger_config_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab
--principal=kudu/127.2.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:40265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14961
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:56:34.041549 14961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:56:34.041786 14961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:56:34.046424 14966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:56:34.046500 14969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:56:34.046435 14967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:56:34.046929 14961 server_base.cc:1034] running on GCE node
I20260928 21:56:34.047188 14961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:56:34.047535 14961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:56:34.048713 14961 hybrid_clock.cc:648] HybridClock initialized: now 1790632594048696 us; error 32 us; skew 500 ppm
Sep 28 21:56:34 dist-test-slave-k6lz krb5kdc[14517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632594, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:56:34.055677 14961 init.cc:377] Logged in from keytab as kudu/127.2.182.126@KRBTEST.COM (short username kudu)
I20260928 21:56:34.056147 14961 webserver.cc:460] Webserver started at http://127.2.182.126:44091/ using document root <none> and password file <none>
I20260928 21:56:34.056423 14961 fs_manager.cc:359] Metadata directory not provided
I20260928 21:56:34.056465 14961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:56:34.056583 14961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 21:56:34 dist-test-slave-k6lz krb5kdc[14517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632594, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.126@KRBTEST.COM for HTTP/127.2.182.84@KRBTEST.COM
I20260928 21:56:34.102272 14961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/master-0/data/instance:
uuid: "23836da853b24db6bb832e4f208a0734"
format_stamp: "Formatted at 2026-09-28 21:56:34 on dist-test-slave-k6lz"
server_key: "5bea734f297a61cd1c8c560ef47b8161"
server_key_iv: "050fb9aef9bf4e8364cda27c7c7831c6"
server_key_version: "kuduclusterkey@0"
I20260928 21:56:34.102640 14961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/master-0/wal/instance:
uuid: "23836da853b24db6bb832e4f208a0734"
format_stamp: "Formatted at 2026-09-28 21:56:34 on dist-test-slave-k6lz"
server_key: "5bea734f297a61cd1c8c560ef47b8161"
server_key_iv: "050fb9aef9bf4e8364cda27c7c7831c6"
server_key_version: "kuduclusterkey@0"
I20260928 21:56:34.104223 14961 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 21:56:34.120283 14976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:56:34.120467 14961 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 21:56:34.120566 14961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/master-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/master-0/wal
uuid: "23836da853b24db6bb832e4f208a0734"
format_stamp: "Formatted at 2026-09-28 21:56:34 on dist-test-slave-k6lz"
server_key: "5bea734f297a61cd1c8c560ef47b8161"
server_key_iv: "050fb9aef9bf4e8364cda27c7c7831c6"
server_key_version: "kuduclusterkey@0"
I20260928 21:56:34.120658 14961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:56:34.133044 14961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:56:34.134847 14961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:56:34.134967 14961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:56:34.139446 14961 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.126:40265
I20260928 21:56:34.139518 15028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.126:40265 every 8 connection(s)
I20260928 21:56:34.139851 14961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
I20260928 21:56:34.140475 15029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:56:34.141171 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 14961
I20260928 21:56:34.141265 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/master-0/wal/instance
I20260928 21:56:34.141449 2777 external_mini_cluster.cc:1428] Setting key 71c0596503504be736a67c24de51ab4b
I20260928 21:56:34.143517 15029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23836da853b24db6bb832e4f208a0734: Bootstrap starting.
I20260928 21:56:34.144191 15029 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23836da853b24db6bb832e4f208a0734: Neither blocks nor log segments found. Creating new log.
I20260928 21:56:34.144469 15029 log.cc:824] T 00000000000000000000000000000000 P 23836da853b24db6bb832e4f208a0734: Log is configured to *not* fsync() on all Append() calls
I20260928 21:56:34.145192 15029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23836da853b24db6bb832e4f208a0734: No bootstrap required, opened a new log
I20260928 21:56:34.147368 15029 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23836da853b24db6bb832e4f208a0734 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23836da853b24db6bb832e4f208a0734" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 40265 } }
I20260928 21:56:34.147524 15029 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23836da853b24db6bb832e4f208a0734 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:56:34.147559 15029 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23836da853b24db6bb832e4f208a0734 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23836da853b24db6bb832e4f208a0734, State: Initialized, Role: FOLLOWER
I20260928 21:56:34.147732 15029 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23836da853b24db6bb832e4f208a0734 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23836da853b24db6bb832e4f208a0734" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 40265 } }
I20260928 21:56:34.147799 15029 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23836da853b24db6bb832e4f208a0734 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:56:34.147831 15029 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23836da853b24db6bb832e4f208a0734 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:56:34.147888 15029 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23836da853b24db6bb832e4f208a0734 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:56:34.148422 15029 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23836da853b24db6bb832e4f208a0734 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23836da853b24db6bb832e4f208a0734" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 40265 } }
I20260928 21:56:34.148525 15029 leader_election.cc:302] T 00000000000000000000000000000000 P 23836da853b24db6bb832e4f208a0734 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23836da853b24db6bb832e4f208a0734; no voters:
I20260928 21:56:34.148702 15029 leader_election.cc:288] T 00000000000000000000000000000000 P 23836da853b24db6bb832e4f208a0734 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:56:34.148748 15034 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23836da853b24db6bb832e4f208a0734 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:56:34.148967 15034 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23836da853b24db6bb832e4f208a0734 [term 1 LEADER]: Becoming Leader. State: Replica: 23836da853b24db6bb832e4f208a0734, State: Running, Role: LEADER
Sep 28 21:56:34 dist-test-slave-k6lz krb5kdc[14517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:56:34.149114 15029 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23836da853b24db6bb832e4f208a0734 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:56:34.149533 15034 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23836da853b24db6bb832e4f208a0734 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23836da853b24db6bb832e4f208a0734" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 40265 } }
I20260928 21:56:34.149914 15034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23836da853b24db6bb832e4f208a0734 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "23836da853b24db6bb832e4f208a0734" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23836da853b24db6bb832e4f208a0734" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 40265 } } }
I20260928 21:56:34.150003 15034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23836da853b24db6bb832e4f208a0734 [sys.catalog]: This master's current role is: LEADER
I20260928 21:56:34.150156 15034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23836da853b24db6bb832e4f208a0734 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23836da853b24db6bb832e4f208a0734. Latest consensus state: current_term: 1 leader_uuid: "23836da853b24db6bb832e4f208a0734" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23836da853b24db6bb832e4f208a0734" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 40265 } } }
I20260928 21:56:34.150239 15034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23836da853b24db6bb832e4f208a0734 [sys.catalog]: This master's current role is: LEADER
I20260928 21:56:34.150422 15029 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 21:56:34.153355 15047 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:56:34.154857 15047 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:56:34.156810 15032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:34.141976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49298 (local address 127.2.182.126:40265)
0928 21:56:34.142165 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 21:56:34.142175 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 21:56:34.142266 (+ 91us) server_negotiation.cc:407] Connection header received
0928 21:56:34.142767 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:56:34.142776 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:56:34.142839 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:56:34.142980 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:56:34.143741 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:34.144963 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:34.146038 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:34.146665 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:34.146791 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:34.149548 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:56:34.149552 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:56:34.149553 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:56:34.149564 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:56:34.154222 (+ 4658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:56:34.154753 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:56:34.154757 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:56:34.154758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:56:34.154935 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:56:34.155160 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:56:34.155165 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:56:34.155165 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:56:34.155737 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:56:34.155904 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 21:56:34.156021 (+ 117us) server_negotiation.cc:299] Negotiation successful
0928 21:56:34.156445 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":94,"threads_started":1}
I20260928 21:56:34.158295 15047 catalog_manager.cc:1347] Generated new cluster ID: 233ff383ca934392b0cdfc57a6eb8add
I20260928 21:56:34.158347 15047 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:56:34.166275 15047 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:56:34.167201 15047 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:56:34.177474 15047 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23836da853b24db6bb832e4f208a0734: Generated new TSK 0
I20260928 21:56:34.177670 15047 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.182.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab.
Entry for principal kudu/127.2.182.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab.
I20260928 21:56:34.879520 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.182.84:37893/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.65:0
--local_ip_for_outbound_sockets=127.2.182.65
--webserver_interface=127.2.182.65
--webserver_port=0
--tserver_master_addrs=127.2.182.126:40265
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab
--principal=kudu/127.2.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:43655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:56:34.999099 15074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:56:34.999408 15074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:56:34.999444 15074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:56:35.001214 15074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:56:35.001289 15074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:56:35.001329 15074 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.65
I20260928 21:56:35.003794 15074 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:43655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.2.182.84:37893/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab
--principal=kudu/127.2.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.182.65
--webserver_port=0
--tserver_master_addrs=127.2.182.126:40265
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15074
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.65
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:56:35.004102 15074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:56:35.004398 15074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:56:35.010041 15079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:56:35.010105 15080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:56:35.010360 15074 server_base.cc:1034] running on GCE node
W20260928 21:56:35.010404 15082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:56:35.010679 15074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:56:35.010995 15074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:56:35.012194 15074 hybrid_clock.cc:648] HybridClock initialized: now 1790632595012180 us; error 33 us; skew 500 ppm
Sep 28 21:56:35 dist-test-slave-k6lz krb5kdc[14517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632595, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:56:35.015548 15074 init.cc:377] Logged in from keytab as kudu/127.2.182.65@KRBTEST.COM (short username kudu)
I20260928 21:56:35.016175 15074 webserver.cc:460] Webserver started at http://127.2.182.65:39741/ using document root <none> and password file <none>
I20260928 21:56:35.016446 15074 fs_manager.cc:359] Metadata directory not provided
I20260928 21:56:35.016505 15074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:56:35.016641 15074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 21:56:35 dist-test-slave-k6lz krb5kdc[14517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632595, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.65@KRBTEST.COM for HTTP/127.2.182.84@KRBTEST.COM
I20260928 21:56:35.029898 15074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-0/data/instance:
uuid: "cfcd102f4a3c41febbbb810e1fd280ef"
format_stamp: "Formatted at 2026-09-28 21:56:35 on dist-test-slave-k6lz"
server_key: "4a56b8b85fcd08b6eb0884dc5c56629f"
server_key_iv: "2dd510fa98b388cd89cdaf132ca52008"
server_key_version: "kuduclusterkey@0"
I20260928 21:56:35.030468 15074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-0/wal/instance:
uuid: "cfcd102f4a3c41febbbb810e1fd280ef"
format_stamp: "Formatted at 2026-09-28 21:56:35 on dist-test-slave-k6lz"
server_key: "4a56b8b85fcd08b6eb0884dc5c56629f"
server_key_iv: "2dd510fa98b388cd89cdaf132ca52008"
server_key_version: "kuduclusterkey@0"
I20260928 21:56:35.033490 15074 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.000s
I20260928 21:56:35.055497 15089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:56:35.055747 15074 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260928 21:56:35.055902 15074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-0/wal
uuid: "cfcd102f4a3c41febbbb810e1fd280ef"
format_stamp: "Formatted at 2026-09-28 21:56:35 on dist-test-slave-k6lz"
server_key: "4a56b8b85fcd08b6eb0884dc5c56629f"
server_key_iv: "2dd510fa98b388cd89cdaf132ca52008"
server_key_version: "kuduclusterkey@0"
I20260928 21:56:35.056104 15074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:56:35.101588 15074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:56:35.103698 15074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:56:35.103848 15074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:56:35.104132 15074 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:56:35.104542 15074 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:56:35.104597 15074 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:56:35.104628 15074 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:56:35.104697 15074 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:56:35.111726 15074 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.65:37697
I20260928 21:56:35.112305 15074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
I20260928 21:56:35.112507 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 15074
I20260928 21:56:35.112604 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-0/wal/instance
I20260928 21:56:35.112816 2777 external_mini_cluster.cc:1428] Setting key 607c929275e7229cc122aef6767c48b5
I20260928 21:56:35.113709 15204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.65:37697 every 8 connection(s)
Sep 28 21:56:35 dist-test-slave-k6lz krb5kdc[14517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632595, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.65@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:56:35.125741 15208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:35.114680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.65:40359 (local address 127.2.182.126:40265)
0928 21:56:35.115448 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0928 21:56:35.115450 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:56:35.115460 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:56:35.115812 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:56:35.115813 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:56:35.115828 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:56:35.115871 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:56:35.116714 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:35.117528 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:35.119095 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:35.119455 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:35.119518 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:35.122194 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:56:35.122196 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:56:35.122197 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:56:35.122206 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:56:35.124162 (+ 1956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:56:35.124611 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:56:35.124613 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:56:35.124613 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:56:35.124696 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:56:35.125002 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:56:35.125004 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:56:35.125004 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:56:35.125160 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:56:35.125231 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 21:56:35.125496 (+ 265us) server_negotiation.cc:299] Negotiation successful
0928 21:56:35.125590 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":738,"thread_start_us":71,"threads_started":1}
I20260928 21:56:35.125770 15207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:35.114373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:40265 (local address 127.2.182.65:40359)
0928 21:56:35.114979 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:35.115021 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 21:56:35.115682 (+ 661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:56:35.116063 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:56:35.116067 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:56:35.116318 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:56:35.116601 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:56:35.116607 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:56:35.117639 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:56:35.117640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:56:35.118991 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:56:35.118994 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:56:35.119604 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:56:35.119605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:56:35.119763 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:35.120372 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:56:35.120383 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:56:35.122070 (+ 1687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:56:35.124269 (+ 2199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:56:35.124279 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:56:35.124282 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:56:35.124529 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:56:35.124788 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:56:35.124789 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:56:35.124789 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:56:35.124928 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:56:35.125231 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:56:35.125235 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:56:35.125367 (+ 132us) client_negotiation.cc:769] Sending connection context
0928 21:56:35.125449 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 21:56:35.125577 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":83,"threads_started":1}
I20260928 21:56:35.126642 15205 heartbeater.cc:347] Connected to a master server at 127.2.182.126:40265
I20260928 21:56:35.126767 15205 heartbeater.cc:464] Registering TS with master...
I20260928 21:56:35.126969 15205 heartbeater.cc:511] Master 127.2.182.126:40265 requested a full tablet report, sending...
I20260928 21:56:35.127599 14993 ts_manager.cc:194] Registered new tserver with Master: cfcd102f4a3c41febbbb810e1fd280ef (127.2.182.65:37697)
I20260928 21:56:35.128557 14993 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.65@KRBTEST.COM'} at 127.2.182.65:40359
WARNING: no policy specified for kudu/127.2.182.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab.
Entry for principal kudu/127.2.182.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab.
I20260928 21:56:35.151432 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.182.84:37893/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.66:0
--local_ip_for_outbound_sockets=127.2.182.66
--webserver_interface=127.2.182.66
--webserver_port=0
--tserver_master_addrs=127.2.182.126:40265
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab
--principal=kudu/127.2.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:43655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:56:35.250735 15211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:56:35.250934 15211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:56:35.250967 15211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:56:35.252334 15211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:56:35.252382 15211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:56:35.252421 15211 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.66
I20260928 21:56:35.254393 15211 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:43655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.2.182.84:37893/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab
--principal=kudu/127.2.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.182.66
--webserver_port=0
--tserver_master_addrs=127.2.182.126:40265
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.66
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:56:35.254650 15211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:56:35.254895 15211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:56:35.260046 15219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:56:35.260017 15218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:56:35.260797 15211 server_base.cc:1034] running on GCE node
W20260928 21:56:35.263502 15221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:56:35.263744 15211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:56:35.264001 15211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:56:35.265152 15211 hybrid_clock.cc:648] HybridClock initialized: now 1790632595265131 us; error 34 us; skew 500 ppm
Sep 28 21:56:35 dist-test-slave-k6lz krb5kdc[14517](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790632595, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 21:56:35 dist-test-slave-k6lz krb5kdc[14517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632595, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:56:35.267726 15211 init.cc:377] Logged in from keytab as kudu/127.2.182.66@KRBTEST.COM (short username kudu)
I20260928 21:56:35.268180 15211 webserver.cc:460] Webserver started at http://127.2.182.66:44545/ using document root <none> and password file <none>
I20260928 21:56:35.268396 15211 fs_manager.cc:359] Metadata directory not provided
I20260928 21:56:35.268446 15211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:56:35.268584 15211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 21:56:35 dist-test-slave-k6lz krb5kdc[14517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632595, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.66@KRBTEST.COM for HTTP/127.2.182.84@KRBTEST.COM
I20260928 21:56:35.289793 15211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-1/data/instance:
uuid: "548b5f40496044d294745fd85228edac"
format_stamp: "Formatted at 2026-09-28 21:56:35 on dist-test-slave-k6lz"
server_key: "e06669f7556c7757993f282a928d0d41"
server_key_iv: "b455ad1e4ee5607117c6ada691a78f65"
server_key_version: "kuduclusterkey@0"
I20260928 21:56:35.290357 15211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-1/wal/instance:
uuid: "548b5f40496044d294745fd85228edac"
format_stamp: "Formatted at 2026-09-28 21:56:35 on dist-test-slave-k6lz"
server_key: "e06669f7556c7757993f282a928d0d41"
server_key_iv: "b455ad1e4ee5607117c6ada691a78f65"
server_key_version: "kuduclusterkey@0"
I20260928 21:56:35.295866 15211 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.000s
I20260928 21:56:35.317610 15228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:56:35.317869 15211 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s
I20260928 21:56:35.317947 15211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-1/wal
uuid: "548b5f40496044d294745fd85228edac"
format_stamp: "Formatted at 2026-09-28 21:56:35 on dist-test-slave-k6lz"
server_key: "e06669f7556c7757993f282a928d0d41"
server_key_iv: "b455ad1e4ee5607117c6ada691a78f65"
server_key_version: "kuduclusterkey@0"
I20260928 21:56:35.318035 15211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:56:35.328876 15211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:56:35.331210 15211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:56:35.331377 15211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:56:35.331568 15211 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:56:35.331888 15211 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:56:35.331928 15211 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:56:35.331954 15211 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:56:35.331971 15211 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:56:35.338439 15211 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.66:37469
I20260928 21:56:35.338963 15211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
I20260928 21:56:35.343631 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 15211
I20260928 21:56:35.343847 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-1/wal/instance
I20260928 21:56:35.344146 2777 external_mini_cluster.cc:1428] Setting key ca4c43dd7f465d7db3150200b8a7276b
I20260928 21:56:35.352905 15341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.66:37469 every 8 connection(s)
Sep 28 21:56:35 dist-test-slave-k6lz krb5kdc[14517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632595, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.66@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:56:35.373394 15208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:35.353958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.66:34597 (local address 127.2.182.126:40265)
0928 21:56:35.354010 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 21:56:35.354013 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:56:35.355599 (+ 1586us) server_negotiation.cc:407] Connection header received
0928 21:56:35.358923 (+ 3324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:56:35.358925 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:56:35.358941 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:56:35.358971 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:56:35.359892 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:35.360713 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:35.365110 (+ 4397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:35.365459 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:35.365511 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:35.370068 (+ 4557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:56:35.370070 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:56:35.370071 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:56:35.370079 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:56:35.371773 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:56:35.372292 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:56:35.372293 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:56:35.372294 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:56:35.372352 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:56:35.372626 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:56:35.372627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:56:35.372627 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:56:35.372766 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:56:35.372813 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 21:56:35.373178 (+ 365us) server_negotiation.cc:299] Negotiation successful
0928 21:56:35.373290 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 21:56:35.374185 15346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:35.353668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:40265 (local address 127.2.182.66:34597)
0928 21:56:35.355447 (+ 1779us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:35.355503 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 21:56:35.358786 (+ 3283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:56:35.359163 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:56:35.359168 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:56:35.359483 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:56:35.359781 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:56:35.359787 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:56:35.360831 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:56:35.360831 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:56:35.362118 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:56:35.362121 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:56:35.365629 (+ 3508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:56:35.365630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:56:35.365790 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:35.368203 (+ 2413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:56:35.368236 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:56:35.369924 (+ 1688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:56:35.371878 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:56:35.371886 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:56:35.371889 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:56:35.372187 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:56:35.372417 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:56:35.372418 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:56:35.372418 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:56:35.372557 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:56:35.372847 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:56:35.372853 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:56:35.373025 (+ 172us) client_negotiation.cc:769] Sending connection context
0928 21:56:35.373720 (+ 695us) client_negotiation.cc:240] Negotiation successful
0928 21:56:35.373899 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1727,"thread_start_us":78,"threads_started":1}
I20260928 21:56:35.374876 15342 heartbeater.cc:347] Connected to a master server at 127.2.182.126:40265
I20260928 21:56:35.374974 15342 heartbeater.cc:464] Registering TS with master...
I20260928 21:56:35.375164 15342 heartbeater.cc:511] Master 127.2.182.126:40265 requested a full tablet report, sending...
I20260928 21:56:35.375614 14993 ts_manager.cc:194] Registered new tserver with Master: 548b5f40496044d294745fd85228edac (127.2.182.66:37469)
I20260928 21:56:35.376097 14993 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.66@KRBTEST.COM'} at 127.2.182.66:34597
WARNING: no policy specified for kudu/127.2.182.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab.
Entry for principal kudu/127.2.182.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab.
I20260928 21:56:35.399065 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.182.84:37893/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.67:0
--local_ip_for_outbound_sockets=127.2.182.67
--webserver_interface=127.2.182.67
--webserver_port=0
--tserver_master_addrs=127.2.182.126:40265
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab
--principal=kudu/127.2.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:43655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:56:35.523967 15348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:56:35.524326 15348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:56:35.524412 15348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:56:35.526258 15348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:56:35.526479 15348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:56:35.526569 15348 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.67
I20260928 21:56:35.528923 15348 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:43655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.2.182.84:37893/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab
--principal=kudu/127.2.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.182.67
--webserver_port=0
--tserver_master_addrs=127.2.182.126:40265
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15348
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.67
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:56:35.529381 15348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:56:35.529704 15348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:56:35.536237 15354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:56:35.536617 15356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:56:35.537144 15353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:56:35.538632 15348 server_base.cc:1034] running on GCE node
I20260928 21:56:35.538870 15348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:56:35.539160 15348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:56:35.540344 15348 hybrid_clock.cc:648] HybridClock initialized: now 1790632595540323 us; error 33 us; skew 500 ppm
Sep 28 21:56:35 dist-test-slave-k6lz krb5kdc[14517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632595, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:56:35.543541 15348 init.cc:377] Logged in from keytab as kudu/127.2.182.67@KRBTEST.COM (short username kudu)
I20260928 21:56:35.544055 15348 webserver.cc:460] Webserver started at http://127.2.182.67:42543/ using document root <none> and password file <none>
I20260928 21:56:35.544281 15348 fs_manager.cc:359] Metadata directory not provided
I20260928 21:56:35.544324 15348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:56:35.544440 15348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 21:56:35 dist-test-slave-k6lz krb5kdc[14517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632595, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.67@KRBTEST.COM for HTTP/127.2.182.84@KRBTEST.COM
I20260928 21:56:35.561038 15348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-2/data/instance:
uuid: "460ba373bf41424fa192b370ac02da3b"
format_stamp: "Formatted at 2026-09-28 21:56:35 on dist-test-slave-k6lz"
server_key: "0e5b9a4a3ce18010c6ee8d1483c2bb3e"
server_key_iv: "637fe114b593ad53e3270d03d213df0b"
server_key_version: "kuduclusterkey@0"
I20260928 21:56:35.562597 15348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-2/wal/instance:
uuid: "460ba373bf41424fa192b370ac02da3b"
format_stamp: "Formatted at 2026-09-28 21:56:35 on dist-test-slave-k6lz"
server_key: "0e5b9a4a3ce18010c6ee8d1483c2bb3e"
server_key_iv: "637fe114b593ad53e3270d03d213df0b"
server_key_version: "kuduclusterkey@0"
I20260928 21:56:35.565824 15348 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.001s
I20260928 21:56:35.581611 15363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:56:35.581863 15348 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:56:35.581988 15348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-2/wal
uuid: "460ba373bf41424fa192b370ac02da3b"
format_stamp: "Formatted at 2026-09-28 21:56:35 on dist-test-slave-k6lz"
server_key: "0e5b9a4a3ce18010c6ee8d1483c2bb3e"
server_key_iv: "637fe114b593ad53e3270d03d213df0b"
server_key_version: "kuduclusterkey@0"
I20260928 21:56:35.582238 15348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:56:35.597055 15348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:56:35.599745 15348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:56:35.599951 15348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:56:35.600241 15348 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:56:35.600658 15348 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:56:35.600744 15348 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:56:35.600808 15348 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:56:35.600859 15348 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:56:35.618450 15348 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.67:39781
I20260928 21:56:35.619381 15348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
I20260928 21:56:35.619983 15476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.67:39781 every 8 connection(s)
I20260928 21:56:35.624960 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 15348
I20260928 21:56:35.625087 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-2/wal/instance
I20260928 21:56:35.625299 2777 external_mini_cluster.cc:1428] Setting key 2471b06016cbaa3aecc4a73ea9e89114
Sep 28 21:56:35 dist-test-slave-k6lz krb5kdc[14517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632595, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.67@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:56:35.632259 15208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:35.620842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.67:57005 (local address 127.2.182.126:40265)
0928 21:56:35.620887 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 21:56:35.620889 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:56:35.621179 (+ 290us) server_negotiation.cc:407] Connection header received
0928 21:56:35.621846 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:56:35.621847 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:56:35.621860 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:56:35.621888 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:56:35.622741 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:35.623535 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:35.625134 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:35.625445 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:35.625486 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:35.629113 (+ 3627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:56:35.629115 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:56:35.629115 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:56:35.629123 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:56:35.630648 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:56:35.631166 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:56:35.631168 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:56:35.631168 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:56:35.631227 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:56:35.631520 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:56:35.631521 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:56:35.631521 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:56:35.631666 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:56:35.631707 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 21:56:35.632075 (+ 368us) server_negotiation.cc:299] Negotiation successful
0928 21:56:35.632162 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 21:56:35.633205 15479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:35.620853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:40265 (local address 127.2.182.67:57005)
0928 21:56:35.621043 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:35.621093 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 21:56:35.621727 (+ 634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:56:35.622109 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:56:35.622113 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:56:35.622355 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:56:35.622622 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:56:35.622628 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:56:35.623684 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:56:35.623685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:56:35.624991 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:56:35.624994 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:56:35.625813 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:56:35.625814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:56:35.626272 (+ 458us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:35.626843 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:56:35.626854 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:56:35.628991 (+ 2137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:56:35.630756 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:56:35.630763 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:56:35.630766 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:56:35.631080 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:56:35.631293 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:56:35.631293 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:56:35.631294 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:56:35.631449 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:56:35.631752 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:56:35.631758 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:56:35.631927 (+ 169us) client_negotiation.cc:769] Sending connection context
0928 21:56:35.632794 (+ 867us) client_negotiation.cc:240] Negotiation successful
0928 21:56:35.632969 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":99,"threads_started":1}
I20260928 21:56:35.633977 15477 heartbeater.cc:347] Connected to a master server at 127.2.182.126:40265
I20260928 21:56:35.634093 15477 heartbeater.cc:464] Registering TS with master...
I20260928 21:56:35.634286 15477 heartbeater.cc:511] Master 127.2.182.126:40265 requested a full tablet report, sending...
I20260928 21:56:35.634706 14993 ts_manager.cc:194] Registered new tserver with Master: 460ba373bf41424fa192b370ac02da3b (127.2.182.67:39781)
I20260928 21:56:35.635171 14993 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.67@KRBTEST.COM'} at 127.2.182.67:57005
I20260928 21:56:35.636876 2777 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:56:35.637615 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 15074
I20260928 21:56:35.645637 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 15211
I20260928 21:56:35.650789 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 15348
I20260928 21:56:35.659500 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 14961
I20260928 21:56:35.666005 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:40265
--webserver_interface=127.2.182.126
--webserver_port=44091
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab
--principal=kudu/127.2.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:43655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.126:40265
--ranger_config_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.182.84:37893/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.182.84:37893/kms with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:56:35.746591 15480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:56:35.746814 15480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:56:35.746845 15480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:56:35.748075 15480 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:56:35.748122 15480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:56:35.748140 15480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:56:35.748155 15480 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:56:35.748170 15480 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:56:35.749756 15480 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:43655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.182.84:37893/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.182.126:40265
--ranger_config_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab
--principal=kudu/127.2.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:40265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.126
--webserver_port=44091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:56:35.749998 15480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:56:35.750198 15480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:56:35.755000 15485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:56:35.755132 15488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:56:35.755138 15480 server_base.cc:1034] running on GCE node
W20260928 21:56:35.755023 15486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:56:35.755614 15480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:56:35.755898 15480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:56:35.757062 15480 hybrid_clock.cc:648] HybridClock initialized: now 1790632595757045 us; error 32 us; skew 500 ppm
Sep 28 21:56:35 dist-test-slave-k6lz krb5kdc[14517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632595, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:56:35.759524 15480 init.cc:377] Logged in from keytab as kudu/127.2.182.126@KRBTEST.COM (short username kudu)
I20260928 21:56:35.759976 15480 webserver.cc:460] Webserver started at http://127.2.182.126:44091/ using document root <none> and password file <none>
I20260928 21:56:35.760190 15480 fs_manager.cc:359] Metadata directory not provided
I20260928 21:56:35.760245 15480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 21:56:35 dist-test-slave-k6lz krb5kdc[14517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632595, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.126@KRBTEST.COM for HTTP/127.2.182.84@KRBTEST.COM
I20260928 21:56:35.822484 15480 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:56:35.823148 15496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:56:35.823385 15480 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:56:35.823499 15480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/master-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/master-0/wal
uuid: "23836da853b24db6bb832e4f208a0734"
format_stamp: "Formatted at 2026-09-28 21:56:34 on dist-test-slave-k6lz"
server_key: "5bea734f297a61cd1c8c560ef47b8161"
server_key_iv: "050fb9aef9bf4e8364cda27c7c7831c6"
server_key_version: "kuduclusterkey@0"
I20260928 21:56:35.823985 15480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:56:35.846017 15480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:56:35.848169 15480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:56:35.848340 15480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:56:35.853343 15480 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.126:40265
I20260928 21:56:35.853375 15548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.126:40265 every 8 connection(s)
I20260928 21:56:35.853832 15480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
I20260928 21:56:35.855118 15549 sys_catalog.cc:263] Verifying existing consensus state
I20260928 21:56:35.855787 15549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23836da853b24db6bb832e4f208a0734: Bootstrap starting.
I20260928 21:56:35.862577 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 15480
I20260928 21:56:35.862931 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.182.84:37893/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.65:37697
--local_ip_for_outbound_sockets=127.2.182.65
--tserver_master_addrs=127.2.182.126:40265
--webserver_port=39741
--webserver_interface=127.2.182.65
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab
--principal=kudu/127.2.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:43655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.182.84:37893/kms with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:56:35.863608 15549 log.cc:824] T 00000000000000000000000000000000 P 23836da853b24db6bb832e4f208a0734: Log is configured to *not* fsync() on all Append() calls
I20260928 21:56:35.865346 15549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23836da853b24db6bb832e4f208a0734: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 21:56:35.865552 15549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23836da853b24db6bb832e4f208a0734: Bootstrap complete.
I20260928 21:56:35.867772 15549 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23836da853b24db6bb832e4f208a0734 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23836da853b24db6bb832e4f208a0734" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 40265 } }
I20260928 21:56:35.868003 15549 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23836da853b24db6bb832e4f208a0734 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23836da853b24db6bb832e4f208a0734, State: Initialized, Role: FOLLOWER
I20260928 21:56:35.868129 15549 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23836da853b24db6bb832e4f208a0734 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23836da853b24db6bb832e4f208a0734" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 40265 } }
I20260928 21:56:35.868196 15549 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23836da853b24db6bb832e4f208a0734 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:56:35.868234 15549 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23836da853b24db6bb832e4f208a0734 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:56:35.868273 15549 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23836da853b24db6bb832e4f208a0734 [term 1 FOLLOWER]: Advancing to term 2
I20260928 21:56:35.869269 15549 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23836da853b24db6bb832e4f208a0734 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23836da853b24db6bb832e4f208a0734" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 40265 } }
I20260928 21:56:35.869364 15549 leader_election.cc:302] T 00000000000000000000000000000000 P 23836da853b24db6bb832e4f208a0734 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23836da853b24db6bb832e4f208a0734; no voters:
I20260928 21:56:35.869525 15549 leader_election.cc:288] T 00000000000000000000000000000000 P 23836da853b24db6bb832e4f208a0734 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 21:56:35.869571 15553 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23836da853b24db6bb832e4f208a0734 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 21:56:35.869700 15553 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23836da853b24db6bb832e4f208a0734 [term 2 LEADER]: Becoming Leader. State: Replica: 23836da853b24db6bb832e4f208a0734, State: Running, Role: LEADER
I20260928 21:56:35.869850 15553 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23836da853b24db6bb832e4f208a0734 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23836da853b24db6bb832e4f208a0734" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 40265 } }
I20260928 21:56:35.869959 15549 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23836da853b24db6bb832e4f208a0734 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:56:35.870172 15554 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23836da853b24db6bb832e4f208a0734 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "23836da853b24db6bb832e4f208a0734" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23836da853b24db6bb832e4f208a0734" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 40265 } } }
I20260928 21:56:35.870213 15553 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23836da853b24db6bb832e4f208a0734 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23836da853b24db6bb832e4f208a0734. Latest consensus state: current_term: 2 leader_uuid: "23836da853b24db6bb832e4f208a0734" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23836da853b24db6bb832e4f208a0734" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 40265 } } }
I20260928 21:56:35.870265 15554 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23836da853b24db6bb832e4f208a0734 [sys.catalog]: This master's current role is: LEADER
I20260928 21:56:35.870271 15553 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23836da853b24db6bb832e4f208a0734 [sys.catalog]: This master's current role is: LEADER
I20260928 21:56:35.870505 15557 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:56:35.871042 15557 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:56:35.871232 15557 catalog_manager.cc:1259] Loaded cluster ID: 233ff383ca934392b0cdfc57a6eb8add
I20260928 21:56:35.871310 15557 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:56:35.871440 15549 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 21:56:35.874789 15557 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:56:35.875262 15557 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 23836da853b24db6bb832e4f208a0734: Loaded TSK: 0
I20260928 21:56:35.875582 15557 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 21:56:35.949844 15552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:56:35.950040 15552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:56:35.950070 15552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:56:35.951530 15552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:56:35.951591 15552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:56:35.951629 15552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.65
I20260928 21:56:35.953322 15552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:43655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.182.84:37893/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab
--principal=kudu/127.2.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.65:37697
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.182.65
--webserver_port=39741
--tserver_master_addrs=127.2.182.126:40265
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.65
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:56:35.953576 15552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:56:35.953824 15552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:56:35.958987 15587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:56:35.959146 15589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:56:35.959372 15586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:56:35.959429 15552 server_base.cc:1034] running on GCE node
I20260928 21:56:35.959699 15552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:56:35.959947 15552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:56:35.961165 15552 hybrid_clock.cc:648] HybridClock initialized: now 1790632595961146 us; error 32 us; skew 500 ppm
Sep 28 21:56:35 dist-test-slave-k6lz krb5kdc[14517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632595, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:56:35.963650 15552 init.cc:377] Logged in from keytab as kudu/127.2.182.65@KRBTEST.COM (short username kudu)
I20260928 21:56:35.964172 15552 webserver.cc:460] Webserver started at http://127.2.182.65:39741/ using document root <none> and password file <none>
I20260928 21:56:35.964385 15552 fs_manager.cc:359] Metadata directory not provided
I20260928 21:56:35.964433 15552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 21:56:35 dist-test-slave-k6lz krb5kdc[14517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632595, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.65@KRBTEST.COM for HTTP/127.2.182.84@KRBTEST.COM
I20260928 21:56:35.984499 15552 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:56:35.985358 15597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:56:35.985494 15552 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:56:35.985569 15552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-0/wal
uuid: "cfcd102f4a3c41febbbb810e1fd280ef"
format_stamp: "Formatted at 2026-09-28 21:56:35 on dist-test-slave-k6lz"
server_key: "4a56b8b85fcd08b6eb0884dc5c56629f"
server_key_iv: "2dd510fa98b388cd89cdaf132ca52008"
server_key_version: "kuduclusterkey@0"
I20260928 21:56:35.985931 15552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:56:36.007779 15552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:56:36.010305 15552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:56:36.010545 15552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:56:36.010830 15552 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:56:36.011225 15552 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:56:36.011298 15552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:56:36.011387 15552 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:56:36.011430 15552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:56:36.019224 15552 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.65:37697
I20260928 21:56:36.020046 15552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
I20260928 21:56:36.025044 15710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.65:37697 every 8 connection(s)
I20260928 21:56:36.028409 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 15552
I20260928 21:56:36.028743 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.182.84:37893/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.66:37469
--local_ip_for_outbound_sockets=127.2.182.66
--tserver_master_addrs=127.2.182.126:40265
--webserver_port=44545
--webserver_interface=127.2.182.66
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab
--principal=kudu/127.2.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:43655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.182.84:37893/kms with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 21:56:36 dist-test-slave-k6lz krb5kdc[14517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632595, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.65@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:56:36.049480 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:36.031960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:40265 (local address 127.2.182.65:47075)
0928 21:56:36.036700 (+ 4740us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:36.036769 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0928 21:56:36.037479 (+ 710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:56:36.038251 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:56:36.038256 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:56:36.038516 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:56:36.038824 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:56:36.038831 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:56:36.040693 (+ 1862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:56:36.040694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:56:36.042007 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:56:36.042011 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:56:36.042903 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:56:36.042904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:56:36.043057 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:36.043668 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:56:36.043680 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:56:36.045351 (+ 1671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:56:36.047539 (+ 2188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:56:36.047546 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:56:36.047549 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:56:36.047821 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:56:36.048123 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:56:36.048124 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:56:36.048124 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:56:36.048258 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:56:36.048743 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:56:36.048750 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:56:36.048904 (+ 154us) client_negotiation.cc:769] Sending connection context
0928 21:56:36.049000 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 21:56:36.049171 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4678,"thread_start_us":106,"threads_started":1}
I20260928 21:56:36.050403 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:36.032379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.65:47075 (local address 127.2.182.126:40265)
0928 21:56:36.037747 (+ 5368us) server_negotiation.cc:206] Beginning negotiation
0928 21:56:36.037757 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 21:56:36.037799 (+ 42us) server_negotiation.cc:407] Connection header received
0928 21:56:36.037949 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:56:36.037956 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:56:36.038025 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:56:36.038974 (+ 949us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:56:36.039076 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:36.040562 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:36.042139 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:36.042775 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:36.043377 (+ 602us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:36.045513 (+ 2136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:56:36.045518 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:56:36.045518 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:56:36.045531 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:56:36.047441 (+ 1910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:56:36.047906 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:56:36.047909 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:56:36.047910 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:56:36.048042 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:56:36.048337 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:56:36.048337 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:56:36.048338 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:56:36.048635 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:56:36.049987 (+ 1352us) server_negotiation.cc:1035] Waiting for connection context
0928 21:56:36.050078 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 21:56:36.050201 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5291,"thread_start_us":91,"threads_started":1}
I20260928 21:56:36.051424 15711 heartbeater.cc:347] Connected to a master server at 127.2.182.126:40265
I20260928 21:56:36.051537 15711 heartbeater.cc:464] Registering TS with master...
I20260928 21:56:36.051751 15711 heartbeater.cc:511] Master 127.2.182.126:40265 requested a full tablet report, sending...
W20260928 21:56:36.052620 15711 heartbeater.cc:644] Failed to heartbeat to 127.2.182.126:40265 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 21:56:36.052690 15711 heartbeater.cc:464] Registering TS with master...
I20260928 21:56:36.052752 15711 heartbeater.cc:511] Master 127.2.182.126:40265 requested a full tablet report, sending...
I20260928 21:56:36.053179 15711 heartbeater.cc:464] Registering TS with master...
I20260928 21:56:36.053254 15711 heartbeater.cc:511] Master 127.2.182.126:40265 requested a full tablet report, sending...
W20260928 21:56:36.053601 15711 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 21:56:36.139015 15713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:56:36.139228 15713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:56:36.139259 15713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:56:36.140585 15713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:56:36.140633 15713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:56:36.140668 15713 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.66
I20260928 21:56:36.142160 15713 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:43655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.182.84:37893/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab
--principal=kudu/127.2.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.66:37469
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.182.66
--webserver_port=44545
--tserver_master_addrs=127.2.182.126:40265
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.66
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:56:36.142395 15713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:56:36.142628 15713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:56:36.149061 15720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:56:36.149083 15721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:56:36.149061 15723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:56:36.149818 15713 server_base.cc:1034] running on GCE node
I20260928 21:56:36.150043 15713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:56:36.150323 15713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:56:36.151489 15713 hybrid_clock.cc:648] HybridClock initialized: now 1790632596151470 us; error 27 us; skew 500 ppm
Sep 28 21:56:36 dist-test-slave-k6lz krb5kdc[14517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632596, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:56:36.154654 15713 init.cc:377] Logged in from keytab as kudu/127.2.182.66@KRBTEST.COM (short username kudu)
I20260928 21:56:36.155215 15713 webserver.cc:460] Webserver started at http://127.2.182.66:44545/ using document root <none> and password file <none>
I20260928 21:56:36.155498 15713 fs_manager.cc:359] Metadata directory not provided
I20260928 21:56:36.155568 15713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 21:56:36 dist-test-slave-k6lz krb5kdc[14517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632596, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.66@KRBTEST.COM for HTTP/127.2.182.84@KRBTEST.COM
I20260928 21:56:36.181218 15713 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260928 21:56:36.182331 15731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:56:36.182539 15713 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 21:56:36.182639 15713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-1/wal
uuid: "548b5f40496044d294745fd85228edac"
format_stamp: "Formatted at 2026-09-28 21:56:35 on dist-test-slave-k6lz"
server_key: "e06669f7556c7757993f282a928d0d41"
server_key_iv: "b455ad1e4ee5607117c6ada691a78f65"
server_key_version: "kuduclusterkey@0"
I20260928 21:56:36.183116 15713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:56:36.200038 15713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:56:36.202418 15713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:56:36.202584 15713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:56:36.202875 15713 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:56:36.203269 15713 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:56:36.203322 15713 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:56:36.203382 15713 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:56:36.203398 15713 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:56:36.210562 15713 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.66:37469
I20260928 21:56:36.211025 15713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
I20260928 21:56:36.211062 15844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.66:37469 every 8 connection(s)
I20260928 21:56:36.212621 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 15713
I20260928 21:56:36.212889 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.182.84:37893/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.67:39781
--local_ip_for_outbound_sockets=127.2.182.67
--tserver_master_addrs=127.2.182.126:40265
--webserver_port=42543
--webserver_interface=127.2.182.67
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab
--principal=kudu/127.2.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:43655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.182.84:37893/kms with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 21:56:36 dist-test-slave-k6lz krb5kdc[14517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632596, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.66@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:56:36.224895 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:36.212292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.66:52251 (local address 127.2.182.126:40265)
0928 21:56:36.212342 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 21:56:36.212344 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:56:36.212355 (+ 11us) server_negotiation.cc:407] Connection header received
0928 21:56:36.213045 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:56:36.213047 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:56:36.213060 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:56:36.213092 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:56:36.213912 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:36.214718 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:36.216252 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:36.216535 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:36.216596 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:36.219881 (+ 3285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:56:36.219883 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:56:36.219883 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:56:36.219891 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:56:36.221529 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:56:36.223830 (+ 2301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:56:36.223831 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:56:36.223831 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:56:36.223892 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:56:36.224157 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:56:36.224158 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:56:36.224158 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:56:36.224294 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:56:36.224342 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 21:56:36.224700 (+ 358us) server_negotiation.cc:299] Negotiation successful
0928 21:56:36.224795 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 21:56:36.225731 15847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:36.211985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:40265 (local address 127.2.182.66:52251)
0928 21:56:36.212174 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:36.212257 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0928 21:56:36.212886 (+ 629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:56:36.213293 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:56:36.213298 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:56:36.213529 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:56:36.213802 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:56:36.213808 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:56:36.214849 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:56:36.214850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:56:36.216136 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:56:36.216139 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:56:36.216707 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:56:36.216709 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:56:36.216954 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:36.217713 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:56:36.217722 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:56:36.219763 (+ 2041us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:56:36.223394 (+ 3631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:56:36.223403 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:56:36.223406 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:56:36.223730 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:56:36.223959 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:56:36.223959 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:56:36.223960 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:56:36.224088 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:56:36.224371 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:56:36.224376 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:56:36.224539 (+ 163us) client_negotiation.cc:769] Sending connection context
0928 21:56:36.225317 (+ 778us) client_negotiation.cc:240] Negotiation successful
0928 21:56:36.225488 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":89,"threads_started":1}
I20260928 21:56:36.226387 15845 heartbeater.cc:347] Connected to a master server at 127.2.182.126:40265
I20260928 21:56:36.226482 15845 heartbeater.cc:464] Registering TS with master...
I20260928 21:56:36.226681 15845 heartbeater.cc:511] Master 127.2.182.126:40265 requested a full tablet report, sending...
W20260928 21:56:36.227392 15845 heartbeater.cc:644] Failed to heartbeat to 127.2.182.126:40265 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 21:56:36.227458 15845 heartbeater.cc:464] Registering TS with master...
I20260928 21:56:36.227519 15845 heartbeater.cc:511] Master 127.2.182.126:40265 requested a full tablet report, sending...
I20260928 21:56:36.227909 15845 heartbeater.cc:464] Registering TS with master...
I20260928 21:56:36.227984 15845 heartbeater.cc:511] Master 127.2.182.126:40265 requested a full tablet report, sending...
W20260928 21:56:36.228318 15845 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 21:56:36.319247 15848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:56:36.319623 15848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:56:36.319703 15848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:56:36.321373 15848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:56:36.321425 15848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:56:36.321465 15848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.67
I20260928 21:56:36.323231 15848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:43655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.182.84:37893/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab
--principal=kudu/127.2.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.67:39781
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.182.67
--webserver_port=42543
--tserver_master_addrs=127.2.182.126:40265
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.67
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:56:36.323515 15848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:56:36.323863 15848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:56:36.329843 15854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:56:36.330030 15856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:56:36.330330 15853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:56:36.330694 15848 server_base.cc:1034] running on GCE node
I20260928 21:56:36.330947 15848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:56:36.331286 15848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:56:36.332616 15848 hybrid_clock.cc:648] HybridClock initialized: now 1790632596332601 us; error 25 us; skew 500 ppm
Sep 28 21:56:36 dist-test-slave-k6lz krb5kdc[14517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632596, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:56:36.335220 15848 init.cc:377] Logged in from keytab as kudu/127.2.182.67@KRBTEST.COM (short username kudu)
I20260928 21:56:36.335709 15848 webserver.cc:460] Webserver started at http://127.2.182.67:42543/ using document root <none> and password file <none>
I20260928 21:56:36.335932 15848 fs_manager.cc:359] Metadata directory not provided
I20260928 21:56:36.335991 15848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 21:56:36 dist-test-slave-k6lz krb5kdc[14517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632596, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.67@KRBTEST.COM for HTTP/127.2.182.84@KRBTEST.COM
I20260928 21:56:36.354772 15848 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:56:36.355931 15864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:56:36.356129 15848 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 21:56:36.356220 15848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-2/wal
uuid: "460ba373bf41424fa192b370ac02da3b"
format_stamp: "Formatted at 2026-09-28 21:56:35 on dist-test-slave-k6lz"
server_key: "0e5b9a4a3ce18010c6ee8d1483c2bb3e"
server_key_iv: "637fe114b593ad53e3270d03d213df0b"
server_key_version: "kuduclusterkey@0"
I20260928 21:56:36.356607 15848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:56:36.367049 15848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:56:36.369961 15848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:56:36.370188 15848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:56:36.370831 15848 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:56:36.371251 15848 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:56:36.371374 15848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:56:36.371454 15848 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:56:36.371510 15848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:56:36.378371 15848 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.67:39781
I20260928 21:56:36.378463 15978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.67:39781 every 8 connection(s)
I20260928 21:56:36.378793 15848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
Sep 28 21:56:36 dist-test-slave-k6lz krb5kdc[14517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632596, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.67@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:56:36.386454 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 15848
I20260928 21:56:36.389034 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:36.379646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.67:41933 (local address 127.2.182.126:40265)
0928 21:56:36.379708 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 21:56:36.379710 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:56:36.379934 (+ 224us) server_negotiation.cc:407] Connection header received
0928 21:56:36.380415 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:56:36.380417 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:56:36.380429 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:56:36.380515 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:56:36.381012 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:36.381741 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:36.382732 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:36.383023 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:36.383106 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:36.385544 (+ 2438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:56:36.385546 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:56:36.385546 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:56:36.385554 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:56:36.387223 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:56:36.387862 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:56:36.387864 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:56:36.387864 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:56:36.387931 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:56:36.388276 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:56:36.388277 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:56:36.388277 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:56:36.388418 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:56:36.388460 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 21:56:36.388858 (+ 398us) server_negotiation.cc:299] Negotiation successful
0928 21:56:36.388942 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 21:56:36.389714 15981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:36.379608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:40265 (local address 127.2.182.67:41933)
0928 21:56:36.379795 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:36.379852 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 21:56:36.380320 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:56:36.380567 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:56:36.380570 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:56:36.380733 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:56:36.380928 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:56:36.380932 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:56:36.381830 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:56:36.381830 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:56:36.382633 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:56:36.382636 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:56:36.383129 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:56:36.383130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:56:36.383282 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:36.383855 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:56:36.383864 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:56:36.385419 (+ 1555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:56:36.387306 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:56:36.387315 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:56:36.387318 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:56:36.387765 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:56:36.388044 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:56:36.388045 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:56:36.388045 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:56:36.388196 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:56:36.388507 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:56:36.388514 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:56:36.388696 (+ 182us) client_negotiation.cc:769] Sending connection context
0928 21:56:36.389286 (+ 590us) client_negotiation.cc:240] Negotiation successful
0928 21:56:36.389472 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":74,"threads_started":1}
I20260928 21:56:36.390249 15979 heartbeater.cc:347] Connected to a master server at 127.2.182.126:40265
I20260928 21:56:36.390360 15979 heartbeater.cc:464] Registering TS with master...
I20260928 21:56:36.390563 15979 heartbeater.cc:511] Master 127.2.182.126:40265 requested a full tablet report, sending...
W20260928 21:56:36.391258 15979 heartbeater.cc:644] Failed to heartbeat to 127.2.182.126:40265 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 21:56:36.391366 15979 heartbeater.cc:464] Registering TS with master...
I20260928 21:56:36.391430 15979 heartbeater.cc:511] Master 127.2.182.126:40265 requested a full tablet report, sending...
I20260928 21:56:36.391772 15979 heartbeater.cc:464] Registering TS with master...
I20260928 21:56:36.391844 15979 heartbeater.cc:511] Master 127.2.182.126:40265 requested a full tablet report, sending...
W20260928 21:56:36.392181 15979 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 21:56:36.395131 15982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:36.386842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49312 (local address 127.2.182.126:40265)
0928 21:56:36.387649 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0928 21:56:36.387652 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:56:36.387659 (+ 7us) server_negotiation.cc:407] Connection header received
0928 21:56:36.387678 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:56:36.387678 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:56:36.387689 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:56:36.388615 (+ 926us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:56:36.388650 (+ 35us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:36.389656 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:36.391236 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:36.391530 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:36.391608 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:36.392844 (+ 1236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:56:36.392845 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:56:36.392845 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:56:36.392852 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:56:36.394091 (+ 1239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:56:36.394388 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:56:36.394389 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:56:36.394389 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:56:36.394436 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:56:36.394597 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:56:36.394598 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:56:36.394598 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:56:36.394696 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:56:36.394862 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 21:56:36.394896 (+ 34us) server_negotiation.cc:299] Negotiation successful
0928 21:56:36.394970 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":787,"thread_start_us":63,"threads_started":1}
Sep 28 21:56:37 dist-test-slave-k6lz krb5kdc[14517](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790632597, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:56:37.053784 15711 heartbeater.cc:464] Registering TS with master...
I20260928 21:56:37.053946 15711 heartbeater.cc:511] Master 127.2.182.126:40265 requested a full tablet report, sending...
I20260928 21:56:37.054330 15513 ts_manager.cc:194] Registered new tserver with Master: cfcd102f4a3c41febbbb810e1fd280ef (127.2.182.65:37697)
I20260928 21:56:37.055368 15513 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.65@KRBTEST.COM'} at 127.2.182.65:47075
I20260928 21:56:37.228484 15845 heartbeater.cc:464] Registering TS with master...
I20260928 21:56:37.228660 15845 heartbeater.cc:511] Master 127.2.182.126:40265 requested a full tablet report, sending...
I20260928 21:56:37.229064 15513 ts_manager.cc:194] Registered new tserver with Master: 548b5f40496044d294745fd85228edac (127.2.182.66:37469)
I20260928 21:56:37.229614 15513 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.66@KRBTEST.COM'} at 127.2.182.66:52251
Sep 28 21:56:37 dist-test-slave-k6lz krb5kdc[14517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632597, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.126@KRBTEST.COM for HTTP/127.2.182.84@KRBTEST.COM
I20260928 21:56:37.392367 15979 heartbeater.cc:464] Registering TS with master...
I20260928 21:56:37.392530 15979 heartbeater.cc:511] Master 127.2.182.126:40265 requested a full tablet report, sending...
I20260928 21:56:37.393023 15513 ts_manager.cc:194] Registered new tserver with Master: 460ba373bf41424fa192b370ac02da3b (127.2.182.67:39781)
I20260928 21:56:37.393523 15513 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.67@KRBTEST.COM'} at 127.2.182.67:41933
I20260928 21:56:37.395155 2777 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:56:37.404186 16000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:37.396229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49322 (local address 127.2.182.126:40265)
0928 21:56:37.396371 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 21:56:37.396374 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:56:37.396546 (+ 172us) server_negotiation.cc:407] Connection header received
0928 21:56:37.396610 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:56:37.396611 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:56:37.396624 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:56:37.396699 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:56:37.397327 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:37.398042 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:37.399008 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:37.399283 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:37.399324 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:37.401111 (+ 1787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:56:37.401112 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:56:37.401112 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:56:37.401120 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:56:37.402872 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:56:37.403257 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:56:37.403258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:56:37.403259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:56:37.403323 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:56:37.403559 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:56:37.403560 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:56:37.403560 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:56:37.403712 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:56:37.403791 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 21:56:37.403999 (+ 208us) server_negotiation.cc:299] Negotiation successful
0928 21:56:37.404088 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":82,"threads_started":1}
I20260928 21:56:37.406872 15513 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49322:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:56:37.409734 15513 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:56:37.424049 16008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:37.418961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.66:37469 (local address 127.0.0.1:54642)
0928 21:56:37.419726 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:37.419734 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 21:56:37.419783 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:56:37.420038 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:56:37.420041 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:56:37.420043 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:56:37.420118 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:56:37.420120 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:56:37.421406 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:56:37.421407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:56:37.422652 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:56:37.422655 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:56:37.423727 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:56:37.423727 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:56:37.423806 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:37.423822 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 21:56:37.423891 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 21:56:37.423947 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":751,"thread_start_us":55,"threads_started":1}
I20260928 21:56:37.424624 16011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:37.419152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54642 (local address 127.2.182.66:37469)
0928 21:56:37.419653 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0928 21:56:37.419659 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:56:37.419830 (+ 171us) server_negotiation.cc:407] Connection header received
0928 21:56:37.419866 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:56:37.419868 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:56:37.419883 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:56:37.419942 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:56:37.420191 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:37.421310 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:37.422749 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:37.423551 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:37.423627 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:37.423691 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 21:56:37.424473 (+ 782us) server_negotiation.cc:299] Negotiation successful
0928 21:56:37.424521 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":55,"threads_started":1}
I20260928 21:56:37.424779 16010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:37.419104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36366 (local address 127.2.182.65:37697)
0928 21:56:37.420047 (+ 943us) server_negotiation.cc:206] Beginning negotiation
0928 21:56:37.420050 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:56:37.420061 (+ 11us) server_negotiation.cc:407] Connection header received
0928 21:56:37.420094 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:56:37.420096 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:56:37.420111 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:56:37.420178 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:56:37.420375 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:37.421436 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:37.423034 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:37.423850 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:37.424234 (+ 384us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:37.424306 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 21:56:37.424651 (+ 345us) server_negotiation.cc:299] Negotiation successful
0928 21:56:37.424696 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":910,"thread_start_us":49,"threads_started":1}
I20260928 21:56:37.424576 16009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:37.418750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.65:37697 (local address 127.0.0.1:36366)
0928 21:56:37.419383 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:37.419433 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 21:56:37.419501 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:56:37.420260 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:56:37.420261 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:56:37.420263 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:56:37.420308 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:56:37.420310 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:56:37.421529 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:56:37.421530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:56:37.422941 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:56:37.422944 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:56:37.424329 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:56:37.424330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:56:37.424396 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:37.424399 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:56:37.424446 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 21:56:37.424489 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":81,"threads_started":1}
I20260928 21:56:37.425686 15779 tablet_service.cc:1467] Processing CreateTablet for tablet 267797d4b0d0457893bf7d14cea77b7e (DEFAULT_TABLE table=test-table [id=2796606ff41842deb68d789a36abb402]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:56:37.425700 15636 tablet_service.cc:1467] Processing CreateTablet for tablet 267797d4b0d0457893bf7d14cea77b7e (DEFAULT_TABLE table=test-table [id=2796606ff41842deb68d789a36abb402]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:56:37.426057 15636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 267797d4b0d0457893bf7d14cea77b7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:56:37.426057 15779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 267797d4b0d0457893bf7d14cea77b7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:56:37.429855 16007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:37.418778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.67:39781 (local address 127.0.0.1:58654)
0928 21:56:37.419412 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:37.419583 (+ 171us) client_negotiation.cc:174] Beginning negotiation
0928 21:56:37.419658 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:56:37.425463 (+ 5805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:56:37.425465 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:56:37.425467 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:56:37.425509 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:56:37.425511 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:56:37.427219 (+ 1708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:56:37.427221 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:56:37.428532 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:56:37.428536 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:56:37.429608 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:56:37.429609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:56:37.429679 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:37.429684 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 21:56:37.429724 (+ 40us) client_negotiation.cc:240] Negotiation successful
0928 21:56:37.429767 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":79,"threads_started":1}
I20260928 21:56:37.430346 16012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:37.419234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58654 (local address 127.2.182.67:39781)
0928 21:56:37.425296 (+ 6062us) server_negotiation.cc:206] Beginning negotiation
0928 21:56:37.425301 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:56:37.425313 (+ 12us) server_negotiation.cc:407] Connection header received
0928 21:56:37.425349 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:56:37.425351 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:56:37.425368 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:56:37.425570 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:56:37.425644 (+ 74us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:37.427135 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:37.428641 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:37.429507 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:37.430024 (+ 517us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:37.430117 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 21:56:37.430144 (+ 27us) server_negotiation.cc:299] Negotiation successful
0928 21:56:37.430185 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6023,"thread_start_us":55,"threads_started":1}
I20260928 21:56:37.431201 15905 tablet_service.cc:1467] Processing CreateTablet for tablet 267797d4b0d0457893bf7d14cea77b7e (DEFAULT_TABLE table=test-table [id=2796606ff41842deb68d789a36abb402]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:56:37.431578 15905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 267797d4b0d0457893bf7d14cea77b7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:56:37.431551 16014 tablet_bootstrap.cc:492] T 267797d4b0d0457893bf7d14cea77b7e P cfcd102f4a3c41febbbb810e1fd280ef: Bootstrap starting.
I20260928 21:56:37.432193 16014 tablet_bootstrap.cc:654] T 267797d4b0d0457893bf7d14cea77b7e P cfcd102f4a3c41febbbb810e1fd280ef: Neither blocks nor log segments found. Creating new log.
I20260928 21:56:37.432492 16014 log.cc:824] T 267797d4b0d0457893bf7d14cea77b7e P cfcd102f4a3c41febbbb810e1fd280ef: Log is configured to *not* fsync() on all Append() calls
I20260928 21:56:37.432848 16013 tablet_bootstrap.cc:492] T 267797d4b0d0457893bf7d14cea77b7e P 548b5f40496044d294745fd85228edac: Bootstrap starting.
I20260928 21:56:37.433959 16013 tablet_bootstrap.cc:654] T 267797d4b0d0457893bf7d14cea77b7e P 548b5f40496044d294745fd85228edac: Neither blocks nor log segments found. Creating new log.
I20260928 21:56:37.434264 16013 log.cc:824] T 267797d4b0d0457893bf7d14cea77b7e P 548b5f40496044d294745fd85228edac: Log is configured to *not* fsync() on all Append() calls
I20260928 21:56:37.435230 16013 tablet_bootstrap.cc:492] T 267797d4b0d0457893bf7d14cea77b7e P 548b5f40496044d294745fd85228edac: No bootstrap required, opened a new log
I20260928 21:56:37.435297 16013 ts_tablet_manager.cc:1397] T 267797d4b0d0457893bf7d14cea77b7e P 548b5f40496044d294745fd85228edac: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 21:56:37.435535 16017 tablet_bootstrap.cc:492] T 267797d4b0d0457893bf7d14cea77b7e P 460ba373bf41424fa192b370ac02da3b: Bootstrap starting.
I20260928 21:56:37.436518 16017 tablet_bootstrap.cc:654] T 267797d4b0d0457893bf7d14cea77b7e P 460ba373bf41424fa192b370ac02da3b: Neither blocks nor log segments found. Creating new log.
I20260928 21:56:37.436894 16017 log.cc:824] T 267797d4b0d0457893bf7d14cea77b7e P 460ba373bf41424fa192b370ac02da3b: Log is configured to *not* fsync() on all Append() calls
I20260928 21:56:37.437925 16013 raft_consensus.cc:348] T 267797d4b0d0457893bf7d14cea77b7e P 548b5f40496044d294745fd85228edac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcd102f4a3c41febbbb810e1fd280ef" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 37697 } } peers { permanent_uuid: "460ba373bf41424fa192b370ac02da3b" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 39781 } } peers { permanent_uuid: "548b5f40496044d294745fd85228edac" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37469 } }
I20260928 21:56:37.438112 16013 raft_consensus.cc:374] T 267797d4b0d0457893bf7d14cea77b7e P 548b5f40496044d294745fd85228edac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:56:37.438143 16013 raft_consensus.cc:729] T 267797d4b0d0457893bf7d14cea77b7e P 548b5f40496044d294745fd85228edac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548b5f40496044d294745fd85228edac, State: Initialized, Role: FOLLOWER
I20260928 21:56:37.438621 16014 tablet_bootstrap.cc:492] T 267797d4b0d0457893bf7d14cea77b7e P cfcd102f4a3c41febbbb810e1fd280ef: No bootstrap required, opened a new log
I20260928 21:56:37.438692 16014 ts_tablet_manager.cc:1397] T 267797d4b0d0457893bf7d14cea77b7e P cfcd102f4a3c41febbbb810e1fd280ef: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260928 21:56:37.441293 16014 raft_consensus.cc:348] T 267797d4b0d0457893bf7d14cea77b7e P cfcd102f4a3c41febbbb810e1fd280ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcd102f4a3c41febbbb810e1fd280ef" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 37697 } } peers { permanent_uuid: "460ba373bf41424fa192b370ac02da3b" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 39781 } } peers { permanent_uuid: "548b5f40496044d294745fd85228edac" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37469 } }
I20260928 21:56:37.441478 16014 raft_consensus.cc:374] T 267797d4b0d0457893bf7d14cea77b7e P cfcd102f4a3c41febbbb810e1fd280ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:56:37.441516 16014 raft_consensus.cc:729] T 267797d4b0d0457893bf7d14cea77b7e P cfcd102f4a3c41febbbb810e1fd280ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfcd102f4a3c41febbbb810e1fd280ef, State: Initialized, Role: FOLLOWER
I20260928 21:56:37.441656 16013 consensus_queue.cc:260] T 267797d4b0d0457893bf7d14cea77b7e P 548b5f40496044d294745fd85228edac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcd102f4a3c41febbbb810e1fd280ef" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 37697 } } peers { permanent_uuid: "460ba373bf41424fa192b370ac02da3b" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 39781 } } peers { permanent_uuid: "548b5f40496044d294745fd85228edac" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37469 } }
I20260928 21:56:37.441917 16014 consensus_queue.cc:260] T 267797d4b0d0457893bf7d14cea77b7e P cfcd102f4a3c41febbbb810e1fd280ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcd102f4a3c41febbbb810e1fd280ef" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 37697 } } peers { permanent_uuid: "460ba373bf41424fa192b370ac02da3b" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 39781 } } peers { permanent_uuid: "548b5f40496044d294745fd85228edac" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37469 } }
I20260928 21:56:37.442270 16013 ts_tablet_manager.cc:1428] T 267797d4b0d0457893bf7d14cea77b7e P 548b5f40496044d294745fd85228edac: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20260928 21:56:37.442548 16014 ts_tablet_manager.cc:1428] T 267797d4b0d0457893bf7d14cea77b7e P cfcd102f4a3c41febbbb810e1fd280ef: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260928 21:56:37.442322 15845 heartbeater.cc:503] Master 127.2.182.126:40265 was elected leader, sending a full tablet report...
I20260928 21:56:37.442584 15711 heartbeater.cc:503] Master 127.2.182.126:40265 was elected leader, sending a full tablet report...
I20260928 21:56:37.444304 16017 tablet_bootstrap.cc:492] T 267797d4b0d0457893bf7d14cea77b7e P 460ba373bf41424fa192b370ac02da3b: No bootstrap required, opened a new log
I20260928 21:56:37.444388 16017 ts_tablet_manager.cc:1397] T 267797d4b0d0457893bf7d14cea77b7e P 460ba373bf41424fa192b370ac02da3b: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20260928 21:56:37.447129 16017 raft_consensus.cc:348] T 267797d4b0d0457893bf7d14cea77b7e P 460ba373bf41424fa192b370ac02da3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcd102f4a3c41febbbb810e1fd280ef" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 37697 } } peers { permanent_uuid: "460ba373bf41424fa192b370ac02da3b" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 39781 } } peers { permanent_uuid: "548b5f40496044d294745fd85228edac" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37469 } }
I20260928 21:56:37.447320 16017 raft_consensus.cc:374] T 267797d4b0d0457893bf7d14cea77b7e P 460ba373bf41424fa192b370ac02da3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:56:37.447414 16017 raft_consensus.cc:729] T 267797d4b0d0457893bf7d14cea77b7e P 460ba373bf41424fa192b370ac02da3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 460ba373bf41424fa192b370ac02da3b, State: Initialized, Role: FOLLOWER
I20260928 21:56:37.447578 16017 consensus_queue.cc:260] T 267797d4b0d0457893bf7d14cea77b7e P 460ba373bf41424fa192b370ac02da3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcd102f4a3c41febbbb810e1fd280ef" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 37697 } } peers { permanent_uuid: "460ba373bf41424fa192b370ac02da3b" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 39781 } } peers { permanent_uuid: "548b5f40496044d294745fd85228edac" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37469 } }
I20260928 21:56:37.447868 16017 ts_tablet_manager.cc:1428] T 267797d4b0d0457893bf7d14cea77b7e P 460ba373bf41424fa192b370ac02da3b: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 21:56:37.448026 15979 heartbeater.cc:503] Master 127.2.182.126:40265 was elected leader, sending a full tablet report...
W20260928 21:56:37.462342 15846 tablet.cc:2369] T 267797d4b0d0457893bf7d14cea77b7e P 548b5f40496044d294745fd85228edac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:56:37.463490 16019 raft_consensus.cc:482] T 267797d4b0d0457893bf7d14cea77b7e P 548b5f40496044d294745fd85228edac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:56:37.463615 16019 raft_consensus.cc:504] T 267797d4b0d0457893bf7d14cea77b7e P 548b5f40496044d294745fd85228edac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcd102f4a3c41febbbb810e1fd280ef" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 37697 } } peers { permanent_uuid: "460ba373bf41424fa192b370ac02da3b" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 39781 } } peers { permanent_uuid: "548b5f40496044d294745fd85228edac" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37469 } }
I20260928 21:56:37.464053 16019 leader_election.cc:288] T 267797d4b0d0457893bf7d14cea77b7e P 548b5f40496044d294745fd85228edac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cfcd102f4a3c41febbbb810e1fd280ef (127.2.182.65:37697), 460ba373bf41424fa192b370ac02da3b (127.2.182.67:39781)
I20260928 21:56:37.468310 16010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:37.464403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.66:34447 (local address 127.2.182.65:37697)
0928 21:56:37.464446 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 21:56:37.464448 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:56:37.464611 (+ 163us) server_negotiation.cc:407] Connection header received
0928 21:56:37.464706 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:56:37.464706 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:56:37.464728 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:56:37.464755 (+ 27us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:56:37.464967 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:37.465709 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:37.467265 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:37.467911 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:37.467961 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:37.467971 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 21:56:37.468194 (+ 223us) server_negotiation.cc:299] Negotiation successful
0928 21:56:37.468235 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 21:56:37.468321 16012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:37.464469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.66:59717 (local address 127.2.182.67:39781)
0928 21:56:37.464508 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 21:56:37.464510 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:56:37.464845 (+ 335us) server_negotiation.cc:407] Connection header received
0928 21:56:37.464945 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:56:37.464946 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:56:37.464955 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:56:37.464980 (+ 25us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:56:37.465178 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:37.465934 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:37.467313 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:37.467913 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:37.467962 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:37.467971 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 21:56:37.468208 (+ 237us) server_negotiation.cc:299] Negotiation successful
0928 21:56:37.468250 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 21:56:37.468575 16023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:37.464279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.67:39781 (local address 127.2.182.66:59717)
0928 21:56:37.464786 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:37.464796 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:56:37.464871 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:56:37.465039 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:56:37.465040 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:56:37.465041 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:56:37.465109 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:56:37.465110 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:56:37.466035 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:56:37.466036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:56:37.467207 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:56:37.467210 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:56:37.468051 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:56:37.468052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:56:37.468130 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:37.468134 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:56:37.468439 (+ 305us) client_negotiation.cc:240] Negotiation successful
0928 21:56:37.468483 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":84,"threads_started":1}
I20260928 21:56:37.468926 16022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:37.464187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.65:37697 (local address 127.2.182.66:34447)
0928 21:56:37.464553 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:37.464564 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:56:37.464637 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:56:37.464819 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:56:37.464820 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:56:37.464822 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:56:37.464903 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:56:37.464904 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:56:37.465812 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:56:37.465813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:56:37.467157 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:56:37.467160 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:56:37.468049 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:56:37.468050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:56:37.468121 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:37.468125 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:56:37.468789 (+ 664us) client_negotiation.cc:240] Negotiation successful
0928 21:56:37.468841 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":28,"thread_start_us":102,"threads_started":1}
I20260928 21:56:37.468945 15933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "267797d4b0d0457893bf7d14cea77b7e" candidate_uuid: "548b5f40496044d294745fd85228edac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "460ba373bf41424fa192b370ac02da3b" is_pre_election: true
I20260928 21:56:37.469115 15933 raft_consensus.cc:2454] T 267797d4b0d0457893bf7d14cea77b7e P 460ba373bf41424fa192b370ac02da3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548b5f40496044d294745fd85228edac in term 0.
I20260928 21:56:37.469360 15732 leader_election.cc:302] T 267797d4b0d0457893bf7d14cea77b7e P 548b5f40496044d294745fd85228edac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 460ba373bf41424fa192b370ac02da3b, 548b5f40496044d294745fd85228edac; no voters:
I20260928 21:56:37.469403 15665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "267797d4b0d0457893bf7d14cea77b7e" candidate_uuid: "548b5f40496044d294745fd85228edac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfcd102f4a3c41febbbb810e1fd280ef" is_pre_election: true
I20260928 21:56:37.469511 16019 raft_consensus.cc:2781] T 267797d4b0d0457893bf7d14cea77b7e P 548b5f40496044d294745fd85228edac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:56:37.469563 16019 raft_consensus.cc:482] T 267797d4b0d0457893bf7d14cea77b7e P 548b5f40496044d294745fd85228edac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:56:37.469564 15665 raft_consensus.cc:2454] T 267797d4b0d0457893bf7d14cea77b7e P cfcd102f4a3c41febbbb810e1fd280ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548b5f40496044d294745fd85228edac in term 0.
I20260928 21:56:37.469594 16019 raft_consensus.cc:3037] T 267797d4b0d0457893bf7d14cea77b7e P 548b5f40496044d294745fd85228edac [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:56:37.470495 16019 raft_consensus.cc:504] T 267797d4b0d0457893bf7d14cea77b7e P 548b5f40496044d294745fd85228edac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcd102f4a3c41febbbb810e1fd280ef" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 37697 } } peers { permanent_uuid: "460ba373bf41424fa192b370ac02da3b" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 39781 } } peers { permanent_uuid: "548b5f40496044d294745fd85228edac" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37469 } }
I20260928 21:56:37.470636 16019 leader_election.cc:288] T 267797d4b0d0457893bf7d14cea77b7e P 548b5f40496044d294745fd85228edac [CANDIDATE]: Term 1 election: Requested vote from peers cfcd102f4a3c41febbbb810e1fd280ef (127.2.182.65:37697), 460ba373bf41424fa192b370ac02da3b (127.2.182.67:39781)
I20260928 21:56:37.470845 15933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "267797d4b0d0457893bf7d14cea77b7e" candidate_uuid: "548b5f40496044d294745fd85228edac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "460ba373bf41424fa192b370ac02da3b"
I20260928 21:56:37.470930 15933 raft_consensus.cc:3037] T 267797d4b0d0457893bf7d14cea77b7e P 460ba373bf41424fa192b370ac02da3b [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:56:37.470935 15665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "267797d4b0d0457893bf7d14cea77b7e" candidate_uuid: "548b5f40496044d294745fd85228edac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfcd102f4a3c41febbbb810e1fd280ef"
I20260928 21:56:37.471014 15665 raft_consensus.cc:3037] T 267797d4b0d0457893bf7d14cea77b7e P cfcd102f4a3c41febbbb810e1fd280ef [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:56:37.471850 15933 raft_consensus.cc:2454] T 267797d4b0d0457893bf7d14cea77b7e P 460ba373bf41424fa192b370ac02da3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548b5f40496044d294745fd85228edac in term 1.
I20260928 21:56:37.471917 15665 raft_consensus.cc:2454] T 267797d4b0d0457893bf7d14cea77b7e P cfcd102f4a3c41febbbb810e1fd280ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548b5f40496044d294745fd85228edac in term 1.
I20260928 21:56:37.472041 15732 leader_election.cc:302] T 267797d4b0d0457893bf7d14cea77b7e P 548b5f40496044d294745fd85228edac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 460ba373bf41424fa192b370ac02da3b, 548b5f40496044d294745fd85228edac; no voters:
I20260928 21:56:37.472118 16019 raft_consensus.cc:2781] T 267797d4b0d0457893bf7d14cea77b7e P 548b5f40496044d294745fd85228edac [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:56:37.472306 16019 raft_consensus.cc:686] T 267797d4b0d0457893bf7d14cea77b7e P 548b5f40496044d294745fd85228edac [term 1 LEADER]: Becoming Leader. State: Replica: 548b5f40496044d294745fd85228edac, State: Running, Role: LEADER
I20260928 21:56:37.472442 16019 consensus_queue.cc:237] T 267797d4b0d0457893bf7d14cea77b7e P 548b5f40496044d294745fd85228edac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcd102f4a3c41febbbb810e1fd280ef" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 37697 } } peers { permanent_uuid: "460ba373bf41424fa192b370ac02da3b" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 39781 } } peers { permanent_uuid: "548b5f40496044d294745fd85228edac" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37469 } }
I20260928 21:56:37.473304 15512 catalog_manager.cc:5495] T 267797d4b0d0457893bf7d14cea77b7e P 548b5f40496044d294745fd85228edac reported cstate change: term changed from 0 to 1, leader changed from <none> to 548b5f40496044d294745fd85228edac (127.2.182.66). New cstate: current_term: 1 leader_uuid: "548b5f40496044d294745fd85228edac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcd102f4a3c41febbbb810e1fd280ef" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 37697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "460ba373bf41424fa192b370ac02da3b" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 39781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "548b5f40496044d294745fd85228edac" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 37469 } health_report { overall_health: HEALTHY } } }
I20260928 21:56:37.481992 16011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:37.478873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54648 (local address 127.2.182.66:37469)
0928 21:56:37.478916 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 21:56:37.478918 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:56:37.478927 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:56:37.478983 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:56:37.478984 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:56:37.479003 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:56:37.479061 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:56:37.479223 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:37.479947 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:37.480951 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:37.481212 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:37.481274 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:37.481455 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:56:37.481591 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:56:37.481672 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 21:56:37.481898 (+ 226us) server_negotiation.cc:299] Negotiation successful
0928 21:56:37.481917 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 21:56:37.484157 15933 raft_consensus.cc:1261] T 267797d4b0d0457893bf7d14cea77b7e P 460ba373bf41424fa192b370ac02da3b [term 1 FOLLOWER]: Refusing update from remote peer 548b5f40496044d294745fd85228edac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:56:37.484182 15665 raft_consensus.cc:1261] T 267797d4b0d0457893bf7d14cea77b7e P cfcd102f4a3c41febbbb810e1fd280ef [term 1 FOLLOWER]: Refusing update from remote peer 548b5f40496044d294745fd85228edac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:56:37.484611 16024 consensus_queue.cc:1035] T 267797d4b0d0457893bf7d14cea77b7e P 548b5f40496044d294745fd85228edac [LEADER]: Connected to new peer: Peer: permanent_uuid: "460ba373bf41424fa192b370ac02da3b" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 39781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:56:37.484722 16019 consensus_queue.cc:1035] T 267797d4b0d0457893bf7d14cea77b7e P 548b5f40496044d294745fd85228edac [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfcd102f4a3c41febbbb810e1fd280ef" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 37697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:56:37.489306 16028 mvcc.cc:205] Tried to move back new op lower bound from 7334431119292190720 to 7334431119247691776. Current Snapshot: MvccSnapshot[applied={T|T < 7334431119292190720}]
I20260928 21:56:37.490260 15512 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:56:37.490357 15512 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:56:37.491077 15512 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 23836da853b24db6bb832e4f208a0734: Sending DeleteTablet for 3 replicas of tablet 267797d4b0d0457893bf7d14cea77b7e
I20260928 21:56:37.491536 15636 tablet_service.cc:1514] Processing DeleteTablet for tablet 267797d4b0d0457893bf7d14cea77b7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:56:37 UTC) from {username='kudu', principal='kudu/127.2.182.126@KRBTEST.COM'} at 127.0.0.1:36366
I20260928 21:56:37.491539 15905 tablet_service.cc:1514] Processing DeleteTablet for tablet 267797d4b0d0457893bf7d14cea77b7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:56:37 UTC) from {username='kudu', principal='kudu/127.2.182.126@KRBTEST.COM'} at 127.0.0.1:58654
I20260928 21:56:37.491932 15779 tablet_service.cc:1514] Processing DeleteTablet for tablet 267797d4b0d0457893bf7d14cea77b7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:56:37 UTC) from {username='kudu', principal='kudu/127.2.182.126@KRBTEST.COM'} at 127.0.0.1:54642
I20260928 21:56:37.492082 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 15552
I20260928 21:56:37.498896 16036 tablet_replica.cc:330] T 267797d4b0d0457893bf7d14cea77b7e P 460ba373bf41424fa192b370ac02da3b: stopping tablet replica
I20260928 21:56:37.499044 16036 raft_consensus.cc:2229] T 267797d4b0d0457893bf7d14cea77b7e P 460ba373bf41424fa192b370ac02da3b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:56:37.499118 16036 raft_consensus.cc:2258] T 267797d4b0d0457893bf7d14cea77b7e P 460ba373bf41424fa192b370ac02da3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:56:37.499271 16037 tablet_replica.cc:330] T 267797d4b0d0457893bf7d14cea77b7e P 548b5f40496044d294745fd85228edac: stopping tablet replica
I20260928 21:56:37.499389 16037 raft_consensus.cc:2229] T 267797d4b0d0457893bf7d14cea77b7e P 548b5f40496044d294745fd85228edac [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:56:37.499491 16037 raft_consensus.cc:2258] T 267797d4b0d0457893bf7d14cea77b7e P 548b5f40496044d294745fd85228edac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:56:37.499989 16037 ts_tablet_manager.cc:1905] T 267797d4b0d0457893bf7d14cea77b7e P 548b5f40496044d294745fd85228edac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:56:37.502094 16036 ts_tablet_manager.cc:1905] T 267797d4b0d0457893bf7d14cea77b7e P 460ba373bf41424fa192b370ac02da3b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 21:56:37.503609 15498 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.182.65:37697 (error 108)
I20260928 21:56:37.504320 16036 ts_tablet_manager.cc:1918] T 267797d4b0d0457893bf7d14cea77b7e P 460ba373bf41424fa192b370ac02da3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:56:37.504397 16036 log.cc:1196] T 267797d4b0d0457893bf7d14cea77b7e P 460ba373bf41424fa192b370ac02da3b: Deleting WAL directory at /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632440055392-2777-0/minicluster-data/ts-2/wal/wals/267797d4b0d0457893bf7d14cea77b7e
I20260928 21:56:37.504637 16036 ts_tablet_manager.cc:1939] T 267797d4b0d0457893bf7d14cea77b7e P 460ba373bf41424fa192b370ac02da3b: Deleting consensus metadata
I20260928 21:56:37.504839 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 15713
I20260928 21:56:37.507869 15497 catalog_manager.cc:4841] TS 460ba373bf41424fa192b370ac02da3b (127.2.182.67:39781): tablet 267797d4b0d0457893bf7d14cea77b7e (table test-table [id=2796606ff41842deb68d789a36abb402]) successfully deleted
I20260928 21:56:37.511518 16007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:37.511318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.65:37697 (local address 127.0.0.1:38126)
0928 21:56:37.511414 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:37.511474 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.65:37697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260928 21:56:37.511663 15498 catalog_manager.cc:4584] TS cfcd102f4a3c41febbbb810e1fd280ef (127.2.182.65:37697): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 267797d4b0d0457893bf7d14cea77b7e: Network error: Client connection negotiation failed: client connection to 127.2.182.65:37697: connect: Connection refused (error 111)
I20260928 21:56:37.512022 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 15848
I20260928 21:56:37.515358 16007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:37.512070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.66:37469 (local address 127.0.0.1:37404)
0928 21:56:37.515241 (+ 3171us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:37.515288 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.66:37469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3141}
I20260928 21:56:37.519088 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 15480
I20260928 21:56:37.523975 2777 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 21:56:37.859120 2777 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 21:56:37.859395 2777 mini_ranger.cc:66] Stopping Ranger...
I20260928 21:56:38.204147 2777 mini_ranger.cc:68] Stopped Ranger
2026-09-28 21:56:38.204 UTC [14547] LOG: received smart shutdown request
2026-09-28 21:56:38.209 UTC [14547] LOG: background worker "logical replication launcher" (PID 14556) exited with exit code 1
2026-09-28 21:56:38.209 UTC [14551] LOG: shutting down
2026-09-28 21:56:38.422 UTC [14547] LOG: database system is shut down
2026-09-28T21:56:38Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (50918 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632440055392-2777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:56:38 dist-test-slave-k6lz krb5kdc[16048](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:56:38 dist-test-slave-k6lz krb5kdc[16048](info): set up 2 sockets
Sep 28 21:56:38 dist-test-slave-k6lz krb5kdc[16048](info): commencing operation
krb5kdc: starting...
W20260928 21:56:39.471005 2777 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Sep 28 21:56:39 dist-test-slave-k6lz krb5kdc[16048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:56:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:56:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab.
Entry for principal kudu/127.2.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab.
I20260928 21:56:39.586256 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632440055392-2777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632440055392-2777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632440055392-2777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:39923
--webserver_interface=127.2.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab
--principal=kudu/127.2.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:37591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.126:39923,127.2.182.125:45213,127.2.182.124:44351
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:56:39.665055 16061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:56:39.665287 16061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:56:39.665315 16061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:56:39.666479 16061 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:56:39.666529 16061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:56:39.666543 16061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:56:39.666553 16061 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:56:39.666564 16061 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:56:39.668109 16061 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:37591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632440055392-2777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632440055392-2777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.126:39923,127.2.182.125:45213,127.2.182.124:44351
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab
--principal=kudu/127.2.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:39923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16061
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632440055392-2777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:56:39.668323 16061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:56:39.668498 16061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:56:39.673581 16067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:56:39.673590 16069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:56:39.673769 16066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:56:39.673949 16061 server_base.cc:1034] running on GCE node
I20260928 21:56:39.674260 16061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:56:39.674543 16061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:56:39.675675 16061 hybrid_clock.cc:648] HybridClock initialized: now 1790632599675657 us; error 37 us; skew 500 ppm
Sep 28 21:56:39 dist-test-slave-k6lz krb5kdc[16048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632599, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:56:39.683837 16061 init.cc:377] Logged in from keytab as kudu/127.2.182.126@KRBTEST.COM (short username kudu)
I20260928 21:56:39.684336 16061 webserver.cc:460] Webserver started at http://127.2.182.126:41491/ using document root <none> and password file <none>
I20260928 21:56:39.684558 16061 fs_manager.cc:359] Metadata directory not provided
I20260928 21:56:39.684607 16061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:56:39.684736 16061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:56:39.685783 16061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632440055392-2777-0/minicluster-data/master-0/data/instance:
uuid: "74162e51f4584b91a89a86bf561a7e97"
format_stamp: "Formatted at 2026-09-28 21:56:39 on dist-test-slave-k6lz"
server_key: "c898cc865b0f50b358e45a18990daf38"
server_key_iv: "047cf3add9c1b1adce1423b837049b03"
server_key_version: "encryptionkey@0"
I20260928 21:56:39.686102 16061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632440055392-2777-0/minicluster-data/master-0/wal/instance:
uuid: "74162e51f4584b91a89a86bf561a7e97"
format_stamp: "Formatted at 2026-09-28 21:56:39 on dist-test-slave-k6lz"
server_key: "c898cc865b0f50b358e45a18990daf38"
server_key_iv: "047cf3add9c1b1adce1423b837049b03"
server_key_version: "encryptionkey@0"
I20260928 21:56:39.687497 16061 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 21:56:39.688331 16076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:56:39.688544 16061 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:56:39.688632 16061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632440055392-2777-0/minicluster-data/master-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632440055392-2777-0/minicluster-data/master-0/wal
uuid: "74162e51f4584b91a89a86bf561a7e97"
format_stamp: "Formatted at 2026-09-28 21:56:39 on dist-test-slave-k6lz"
server_key: "c898cc865b0f50b358e45a18990daf38"
server_key_iv: "047cf3add9c1b1adce1423b837049b03"
server_key_version: "encryptionkey@0"
I20260928 21:56:39.688731 16061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632440055392-2777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632440055392-2777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632440055392-2777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:56:39.707581 16061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:56:39.709491 16061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:56:39.709625 16061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:56:39.713531 16061 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.126:39923
I20260928 21:56:39.713578 16128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.126:39923 every 8 connection(s)
I20260928 21:56:39.713923 16061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
I20260928 21:56:39.714946 16129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:56:39.716603 16129 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.182.126" port: 39923 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:56:39.721210 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 16061
I20260928 21:56:39.721333 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632440055392-2777-0/minicluster-data/master-0/wal/instance
I20260928 21:56:39.721535 2777 external_mini_cluster.cc:1428] Setting key e2b2e6ac71257a9972ce7032b3278512
Sep 28 21:56:39 dist-test-slave-k6lz krb5kdc[16048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632599, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.126@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:56:39.726435 16131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:39.717297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:39923 (local address 127.0.0.1:59056)
0928 21:56:39.717448 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:39.717484 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 21:56:39.717908 (+ 424us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:56:39.718246 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:56:39.718248 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:56:39.718495 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:56:39.718652 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:56:39.718657 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:56:39.719881 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:56:39.719882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:56:39.720628 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:56:39.720630 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:56:39.721214 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:56:39.721216 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:56:39.721297 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:39.721838 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:56:39.721848 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:56:39.723310 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:56:39.725066 (+ 1756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:56:39.725072 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:56:39.725075 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:56:39.725280 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:56:39.725481 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:56:39.725482 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:56:39.725482 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:56:39.725566 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:56:39.725881 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:56:39.725885 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:56:39.725944 (+ 59us) client_negotiation.cc:769] Sending connection context
0928 21:56:39.726023 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 21:56:39.726253 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":90,"thread_start_us":69,"threads_started":1}
I20260928 21:56:39.726433 16132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:39.717316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59056 (local address 127.2.182.126:39923)
0928 21:56:39.717477 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 21:56:39.717487 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 21:56:39.717598 (+ 111us) server_negotiation.cc:407] Connection header received
0928 21:56:39.718069 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:56:39.718071 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:56:39.718089 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:56:39.718176 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:56:39.718774 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:39.719771 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:39.720718 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:39.721064 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:39.721159 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:39.723422 (+ 2263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:56:39.723428 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:56:39.723429 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:56:39.723435 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:56:39.724988 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:56:39.725352 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:56:39.725356 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:56:39.725356 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:56:39.725428 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:56:39.725618 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:56:39.725619 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:56:39.725619 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:56:39.725822 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:56:39.725890 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 21:56:39.726052 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 21:56:39.726181 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":81,"threads_started":1}
I20260928 21:56:39.727295 16129 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.182.125" port: 45213 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:56:39.727653 16131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:39.727548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.125:45213 (local address 127.0.0.1:41296)
0928 21:56:39.727590 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:39.727631 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.125:45213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 21:56:39.727798 16077 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.182.125:45213: connect: Connection refused (error 111)
I20260928 21:56:39.728195 16131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:39.728103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.125:45213 (local address 127.0.0.1:41310)
0928 21:56:39.728148 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:39.728172 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.125:45213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
WARNING: no policy specified for kudu/127.2.182.125@KRBTEST.COM; defaulting to no policy
W20260928 21:56:39.728284 16129 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.182.125:45213: Network error: Client connection negotiation failed: client connection to 127.2.182.125:45213: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.125.keytab.
Entry for principal kudu/127.2.182.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.125.keytab.
I20260928 21:56:39.746380 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632440055392-2777-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632440055392-2777-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632440055392-2777-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632440055392-2777-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.125:45213
--webserver_interface=127.2.182.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.125.keytab
--principal=kudu/127.2.182.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:37591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.126:39923,127.2.182.125:45213,127.2.182.124:44351
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:56:39.777510 16129 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.182.125" port: 45213 } attempt: 1
I20260928 21:56:39.778016 16131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:39.777894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.125:45213 (local address 127.0.0.1:41312)
0928 21:56:39.777950 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:39.777983 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.125:45213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 21:56:39.778412 16131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:39.778323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.125:45213 (local address 127.0.0.1:41320)
0928 21:56:39.778366 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:39.778391 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.125:45213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 21:56:39.778501 16129 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.182.125:45213: Network error: Client connection negotiation failed: client connection to 127.2.182.125:45213: connect: Connection refused (error 111)
W20260928 21:56:39.829368 16136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:56:39.829593 16136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:56:39.829624 16136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:56:39.830828 16136 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:56:39.830885 16136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:56:39.830906 16136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:56:39.830922 16136 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:56:39.830937 16136 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:56:39.832540 16136 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:37591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632440055392-2777-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632440055392-2777-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.126:39923,127.2.182.125:45213,127.2.182.124:44351
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.125.keytab
--principal=kudu/127.2.182.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.125:45213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632440055392-2777-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.182.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632440055392-2777-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:56:39.832804 16136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:56:39.833017 16136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:56:39.837610 16144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:56:39.837680 16141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:56:39.837733 16142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:56:39.838009 16136 server_base.cc:1034] running on GCE node
I20260928 21:56:39.838275 16136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:56:39.838542 16136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:56:39.839716 16136 hybrid_clock.cc:648] HybridClock initialized: now 1790632599839695 us; error 35 us; skew 500 ppm
Sep 28 21:56:39 dist-test-slave-k6lz krb5kdc[16048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632599, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:56:39.842151 16136 init.cc:377] Logged in from keytab as kudu/127.2.182.125@KRBTEST.COM (short username kudu)
I20260928 21:56:39.842669 16136 webserver.cc:460] Webserver started at http://127.2.182.125:39105/ using document root <none> and password file <none>
I20260928 21:56:39.842865 16136 fs_manager.cc:359] Metadata directory not provided
I20260928 21:56:39.842905 16136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:56:39.842991 16136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:56:39.844061 16136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632440055392-2777-0/minicluster-data/master-1/data/instance:
uuid: "d5c5939a108646dbaa66308d20b431f9"
format_stamp: "Formatted at 2026-09-28 21:56:39 on dist-test-slave-k6lz"
server_key: "da0401e9b0fa7911aa8b91796927aeca"
server_key_iv: "1c88c226ea412ff476eeda73d122918e"
server_key_version: "encryptionkey@0"
I20260928 21:56:39.844373 16136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632440055392-2777-0/minicluster-data/master-1/wal/instance:
uuid: "d5c5939a108646dbaa66308d20b431f9"
format_stamp: "Formatted at 2026-09-28 21:56:39 on dist-test-slave-k6lz"
server_key: "da0401e9b0fa7911aa8b91796927aeca"
server_key_iv: "1c88c226ea412ff476eeda73d122918e"
server_key_version: "encryptionkey@0"
I20260928 21:56:39.845633 16136 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 21:56:39.846402 16151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:56:39.846598 16136 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:56:39.846640 16129 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.182.125" port: 45213 } attempt: 2
I20260928 21:56:39.846689 16136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632440055392-2777-0/minicluster-data/master-1/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632440055392-2777-0/minicluster-data/master-1/wal
uuid: "d5c5939a108646dbaa66308d20b431f9"
format_stamp: "Formatted at 2026-09-28 21:56:39 on dist-test-slave-k6lz"
server_key: "da0401e9b0fa7911aa8b91796927aeca"
server_key_iv: "1c88c226ea412ff476eeda73d122918e"
server_key_version: "encryptionkey@0"
I20260928 21:56:39.846760 16136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632440055392-2777-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632440055392-2777-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632440055392-2777-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:56:39.847067 16131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:39.846923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.125:45213 (local address 127.0.0.1:41332)
0928 21:56:39.846997 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:39.847035 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.125:45213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 21:56:39.847463 16131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:39.847352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.125:45213 (local address 127.0.0.1:41334)
0928 21:56:39.847402 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:39.847438 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.125:45213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 21:56:39.847558 16129 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.182.125:45213: Network error: Client connection negotiation failed: client connection to 127.2.182.125:45213: connect: Connection refused (error 111)
I20260928 21:56:39.864741 16136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:56:39.866983 16136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:56:39.867121 16136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:56:39.871003 16136 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.125:45213
I20260928 21:56:39.871100 16203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.125:45213 every 8 connection(s)
I20260928 21:56:39.871464 16136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632440055392-2777-0/minicluster-data/master-1/data/info.pb
I20260928 21:56:39.872434 16204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:56:39.873966 16204 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.182.126" port: 39923 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 21:56:39 dist-test-slave-k6lz krb5kdc[16048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632599, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.125@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:56:39.881387 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 16136
I20260928 21:56:39.881493 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632440055392-2777-0/minicluster-data/master-1/wal/instance
I20260928 21:56:39.881670 2777 external_mini_cluster.cc:1428] Setting key f02e2bc39ad0533b80a1bb53430d84e0
I20260928 21:56:39.883287 16132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:39.874765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59062 (local address 127.2.182.126:39923)
0928 21:56:39.874826 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 21:56:39.874828 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:56:39.875019 (+ 191us) server_negotiation.cc:407] Connection header received
0928 21:56:39.875541 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:56:39.875543 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:56:39.875554 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:56:39.875658 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:56:39.876299 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:39.876829 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:39.877853 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:39.878061 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:39.878156 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:39.880586 (+ 2430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:56:39.880588 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:56:39.880589 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:56:39.880596 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:56:39.881944 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:56:39.882344 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:56:39.882345 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:56:39.882346 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:56:39.882401 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:56:39.882627 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:56:39.882628 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:56:39.882629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:56:39.882759 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:56:39.882815 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 21:56:39.883094 (+ 279us) server_negotiation.cc:299] Negotiation successful
0928 21:56:39.883190 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 21:56:39.883502 16206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:39.874685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:39923 (local address 127.0.0.1:59062)
0928 21:56:39.874893 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:39.874929 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 21:56:39.875444 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:56:39.875759 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:56:39.875763 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:56:39.876012 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:56:39.876222 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:56:39.876226 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:56:39.876938 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:56:39.876939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:56:39.877765 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:56:39.877768 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:56:39.878139 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:56:39.878140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:56:39.878300 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:39.878863 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:56:39.878872 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:56:39.880481 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:56:39.882028 (+ 1547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:56:39.882036 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:56:39.882039 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:56:39.882276 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:56:39.882459 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:56:39.882459 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:56:39.882460 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:56:39.882570 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:56:39.882819 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:56:39.882825 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:56:39.882972 (+ 147us) client_negotiation.cc:769] Sending connection context
0928 21:56:39.883065 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 21:56:39.883259 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":86,"threads_started":1}
I20260928 21:56:39.884040 16204 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.182.125" port: 45213 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 21:56:39 dist-test-slave-k6lz krb5kdc[16048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632599, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.125@KRBTEST.COM for kudu/127.2.182.125@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.182.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.124@KRBTEST.COM" created.
I20260928 21:56:39.891681 16206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:39.884302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.125:45213 (local address 127.0.0.1:41336)
0928 21:56:39.884374 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:39.884392 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 21:56:39.884447 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:56:39.884769 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:56:39.884770 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:56:39.884912 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:56:39.884966 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:56:39.884968 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:56:39.885915 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:56:39.885915 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:56:39.886410 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:56:39.886412 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:56:39.886849 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:56:39.886850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:56:39.886910 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:39.887422 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:56:39.887426 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:56:39.888847 (+ 1421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:56:39.890622 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:56:39.890623 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:56:39.890624 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:56:39.890808 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:56:39.890984 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:56:39.890985 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:56:39.890985 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:56:39.891037 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:56:39.891362 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:56:39.891363 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:56:39.891401 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 21:56:39.891461 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 21:56:39.891578 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 21:56:39.891780 16208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:39.884445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41336 (local address 127.2.182.125:45213)
0928 21:56:39.884630 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 21:56:39.884635 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:56:39.884658 (+ 23us) server_negotiation.cc:407] Connection header received
0928 21:56:39.884679 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:56:39.884681 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:56:39.884695 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:56:39.884760 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:56:39.885030 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:39.885816 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:39.886480 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:39.886786 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:39.886844 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:39.888933 (+ 2089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:56:39.888944 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:56:39.888944 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:56:39.888951 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:56:39.890507 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:56:39.890878 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:56:39.890881 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:56:39.890881 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:56:39.890931 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:56:39.891089 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:56:39.891089 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:56:39.891090 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:56:39.891285 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:56:39.891379 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 21:56:39.891490 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 21:56:39.891669 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":110,"server-negotiator.queue_time_us":156,"thread_start_us":106,"threads_started":1}
I20260928 21:56:39.892287 16204 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.182.124" port: 44351 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:56:39.892621 16206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:39.892506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.124:44351 (local address 127.0.0.1:41766)
0928 21:56:39.892545 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:39.892600 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.124:44351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260928 21:56:39.892795 16154 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.182.124:44351: connect: Connection refused (error 111)
I20260928 21:56:39.893225 16206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:39.893126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.124:44351 (local address 127.0.0.1:41778)
0928 21:56:39.893172 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:39.893200 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.124:44351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 21:56:39.893342 16204 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.182.124:44351: Network error: Client connection negotiation failed: client connection to 127.2.182.124:44351: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.124.keytab.
Entry for principal kudu/127.2.182.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.124.keytab.
I20260928 21:56:39.906802 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632440055392-2777-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632440055392-2777-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632440055392-2777-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632440055392-2777-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.124:44351
--webserver_interface=127.2.182.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.124.keytab
--principal=kudu/127.2.182.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:37591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.126:39923,127.2.182.125:45213,127.2.182.124:44351
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:56:39.938730 16129 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.182.125" port: 45213 } attempt: 3
I20260928 21:56:39.942610 16204 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.182.124" port: 44351 } attempt: 1
I20260928 21:56:39.943001 16206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:39.942894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.124:44351 (local address 127.0.0.1:41794)
0928 21:56:39.942939 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:39.942969 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.124:44351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260928 21:56:39.943385 16206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:39.943273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.124:44351 (local address 127.0.0.1:41796)
0928 21:56:39.943312 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:39.943359 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.124:44351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 21:56:39.943507 16204 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.182.124:44351: Network error: Client connection negotiation failed: client connection to 127.2.182.124:44351: connect: Connection refused (error 111)
Sep 28 21:56:39 dist-test-slave-k6lz krb5kdc[16048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632599, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.126@KRBTEST.COM for kudu/127.2.182.125@KRBTEST.COM
I20260928 21:56:39.946539 16131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:39.939151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.125:45213 (local address 127.0.0.1:41340)
0928 21:56:39.939211 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:39.939224 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:56:39.939326 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:56:39.939510 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:56:39.939510 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:56:39.939672 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:56:39.939755 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:56:39.939757 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:56:39.940724 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:56:39.940725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:56:39.941235 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:56:39.941264 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:56:39.941750 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:56:39.941751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:56:39.941827 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:39.942338 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:56:39.942343 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:56:39.943857 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:56:39.945543 (+ 1686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:56:39.945544 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:56:39.945545 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:56:39.945719 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:56:39.945913 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:56:39.945914 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:56:39.945914 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:56:39.945974 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:56:39.946267 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:56:39.946268 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:56:39.946318 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 21:56:39.946383 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 21:56:39.946470 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 21:56:39.946580 16208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:39.939249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41340 (local address 127.2.182.125:45213)
0928 21:56:39.939314 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 21:56:39.939316 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:56:39.939348 (+ 32us) server_negotiation.cc:407] Connection header received
0928 21:56:39.939410 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:56:39.939411 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:56:39.939425 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:56:39.939502 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:56:39.939833 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:39.940630 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:39.941339 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:39.941669 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:39.941735 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:39.943955 (+ 2220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:56:39.943957 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:56:39.943958 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:56:39.943967 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:56:39.945448 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:56:39.945788 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:56:39.945789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:56:39.945790 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:56:39.945851 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:56:39.946027 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:56:39.946027 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:56:39.946028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:56:39.946160 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:56:39.946224 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 21:56:39.946416 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 21:56:39.946497 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 21:56:39.946976 16129 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.182.124" port: 44351 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:56:39.947362 16131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:39.947218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.124:44351 (local address 127.0.0.1:41804)
0928 21:56:39.947282 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:39.947314 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.124:44351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 21:56:39.947686 16131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:39.947602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.124:44351 (local address 127.0.0.1:41814)
0928 21:56:39.947641 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:39.947666 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.124:44351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 21:56:39.947772 16129 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.182.124:44351: Network error: Client connection negotiation failed: client connection to 127.2.182.124:44351: connect: Connection refused (error 111)
I20260928 21:56:39.978951 16129 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.182.124" port: 44351 } attempt: 1
I20260928 21:56:39.979461 16131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:39.979320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.124:44351 (local address 127.0.0.1:41824)
0928 21:56:39.979399 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:39.979431 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.124:44351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 21:56:39.979876 16131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:39.979789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.124:44351 (local address 127.0.0.1:41826)
0928 21:56:39.979829 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:39.979854 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.124:44351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 21:56:39.979962 16129 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.182.124:44351: Network error: Client connection negotiation failed: client connection to 127.2.182.124:44351: connect: Connection refused (error 111)
W20260928 21:56:39.991405 16211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:56:39.991653 16211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:56:39.991688 16211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:56:39.992991 16211 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:56:39.993049 16211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:56:39.993070 16211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:56:39.993086 16211 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:56:39.993103 16211 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:56:39.994699 16211 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:37591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632440055392-2777-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632440055392-2777-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.126:39923,127.2.182.125:45213,127.2.182.124:44351
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.124.keytab
--principal=kudu/127.2.182.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.124:44351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632440055392-2777-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.182.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632440055392-2777-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:56:39.994956 16211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:56:39.995182 16211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:56:39.999969 16219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:56:39.999963 16217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:56:39.999969 16216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:56:40.000362 16211 server_base.cc:1034] running on GCE node
I20260928 21:56:40.000615 16211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:56:40.000924 16211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:56:40.002101 16211 hybrid_clock.cc:648] HybridClock initialized: now 1790632600002084 us; error 27 us; skew 500 ppm
Sep 28 21:56:40 dist-test-slave-k6lz krb5kdc[16048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632600, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:56:40.004935 16211 init.cc:377] Logged in from keytab as kudu/127.2.182.124@KRBTEST.COM (short username kudu)
I20260928 21:56:40.005415 16211 webserver.cc:460] Webserver started at http://127.2.182.124:34611/ using document root <none> and password file <none>
I20260928 21:56:40.005630 16211 fs_manager.cc:359] Metadata directory not provided
I20260928 21:56:40.005679 16211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:56:40.005805 16211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:56:40.006898 16211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632440055392-2777-0/minicluster-data/master-2/data/instance:
uuid: "4fe8733be2434a29ab1c7cd531d003a4"
format_stamp: "Formatted at 2026-09-28 21:56:40 on dist-test-slave-k6lz"
server_key: "ed87d134efa38ab19c32ef2e4024a671"
server_key_iv: "499b8dbf4ab06c13ef16575c69cc13f9"
server_key_version: "encryptionkey@0"
I20260928 21:56:40.007236 16211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632440055392-2777-0/minicluster-data/master-2/wal/instance:
uuid: "4fe8733be2434a29ab1c7cd531d003a4"
format_stamp: "Formatted at 2026-09-28 21:56:40 on dist-test-slave-k6lz"
server_key: "ed87d134efa38ab19c32ef2e4024a671"
server_key_iv: "499b8dbf4ab06c13ef16575c69cc13f9"
server_key_version: "encryptionkey@0"
I20260928 21:56:40.008587 16211 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:56:40.009472 16226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:56:40.009653 16211 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 21:56:40.009737 16211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632440055392-2777-0/minicluster-data/master-2/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632440055392-2777-0/minicluster-data/master-2/wal
uuid: "4fe8733be2434a29ab1c7cd531d003a4"
format_stamp: "Formatted at 2026-09-28 21:56:40 on dist-test-slave-k6lz"
server_key: "ed87d134efa38ab19c32ef2e4024a671"
server_key_iv: "499b8dbf4ab06c13ef16575c69cc13f9"
server_key_version: "encryptionkey@0"
I20260928 21:56:40.009800 16211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632440055392-2777-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632440055392-2777-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632440055392-2777-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:56:40.011672 16204 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.182.124" port: 44351 } attempt: 2
I20260928 21:56:40.012077 16206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:40.011964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.124:44351 (local address 127.0.0.1:41836)
0928 21:56:40.012017 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:40.012048 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.124:44351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 21:56:40.012454 16206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:40.012316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.124:44351 (local address 127.0.0.1:41850)
0928 21:56:40.012371 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:40.012412 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.124:44351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 21:56:40.012584 16204 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.182.124:44351: Network error: Client connection negotiation failed: client connection to 127.2.182.124:44351: connect: Connection refused (error 111)
I20260928 21:56:40.021320 16211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:56:40.023859 16211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:56:40.023999 16211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:56:40.027757 16211 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.124:44351
I20260928 21:56:40.027792 16278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.124:44351 every 8 connection(s)
I20260928 21:56:40.028163 16211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632440055392-2777-0/minicluster-data/master-2/data/info.pb
I20260928 21:56:40.029157 16279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:56:40.030826 16279 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.182.126" port: 39923 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:56:40.031767 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 16211
I20260928 21:56:40.031878 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632440055392-2777-0/minicluster-data/master-2/wal/instance
I20260928 21:56:40.032054 2777 external_mini_cluster.cc:1428] Setting key c7adfb1ec589a09bb618c5046a0e8c5b
Sep 28 21:56:40 dist-test-slave-k6lz krb5kdc[16048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632600, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.124@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
Sep 28 21:56:40 dist-test-slave-k6lz krb5kdc[16048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.182.124@KRBTEST.COM
I20260928 21:56:40.040225 16132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:40.031613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59072 (local address 127.2.182.126:39923)
0928 21:56:40.031668 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 21:56:40.031671 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:56:40.032009 (+ 338us) server_negotiation.cc:407] Connection header received
0928 21:56:40.032600 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:56:40.032602 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:56:40.032614 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:56:40.032652 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:56:40.033221 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:40.033704 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:40.034711 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:40.034924 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:40.034981 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:40.037258 (+ 2277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:56:40.037260 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:56:40.037260 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:56:40.037268 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:56:40.038703 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:56:40.039097 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:56:40.039098 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:56:40.039098 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:56:40.039239 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:56:40.039527 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:56:40.039528 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:56:40.039528 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:56:40.039665 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:56:40.039732 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 21:56:40.040022 (+ 290us) server_negotiation.cc:299] Negotiation successful
0928 21:56:40.040127 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 21:56:40.040390 16283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:40.032415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41866 (local address 127.2.182.124:44351)
0928 21:56:40.032622 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 21:56:40.032627 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:56:40.032655 (+ 28us) server_negotiation.cc:407] Connection header received
0928 21:56:40.032718 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:56:40.032720 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:56:40.032740 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:56:40.032816 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:56:40.033373 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:40.034204 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:40.034883 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:40.035194 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:40.035251 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:40.037655 (+ 2404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:56:40.037664 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:56:40.037664 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:56:40.037671 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:56:40.039150 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:56:40.039493 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:56:40.039496 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:56:40.039496 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:56:40.039561 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:56:40.039725 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:56:40.039725 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:56:40.039726 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:56:40.039892 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:56:40.040038 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 21:56:40.040151 (+ 113us) server_negotiation.cc:299] Negotiation successful
0928 21:56:40.040214 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":74,"threads_started":1}
I20260928 21:56:40.040393 16281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:40.031600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:39923 (local address 127.0.0.1:59072)
0928 21:56:40.031830 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:40.031914 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0928 21:56:40.032513 (+ 599us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:56:40.032723 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:56:40.032727 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:56:40.032981 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:56:40.033134 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:56:40.033139 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:56:40.033773 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:56:40.033774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:56:40.034624 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:56:40.034626 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:56:40.034993 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:56:40.034994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:56:40.035116 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:40.035644 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:56:40.035654 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:56:40.037110 (+ 1456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:56:40.038778 (+ 1668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:56:40.038784 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:56:40.038786 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:56:40.039035 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:56:40.039312 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:56:40.039313 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:56:40.039313 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:56:40.039460 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:56:40.039726 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:56:40.039731 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:56:40.039884 (+ 153us) client_negotiation.cc:769] Sending connection context
0928 21:56:40.039975 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 21:56:40.040127 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":47,"thread_start_us":97,"threads_started":1}
I20260928 21:56:40.041078 16279 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.182.125" port: 45213 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 21:56:40 dist-test-slave-k6lz krb5kdc[16048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632600, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.124@KRBTEST.COM for kudu/127.2.182.125@KRBTEST.COM
Sep 28 21:56:40 dist-test-slave-k6lz krb5kdc[16048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:56:40.048887 16208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:40.041395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41344 (local address 127.2.182.125:45213)
0928 21:56:40.041444 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 21:56:40.041447 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:56:40.041455 (+ 8us) server_negotiation.cc:407] Connection header received
0928 21:56:40.041482 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:56:40.041483 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:56:40.041493 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:56:40.041552 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:56:40.041727 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:40.042400 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:40.043240 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:40.043617 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:40.043687 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:40.045593 (+ 1906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:56:40.045595 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:56:40.045595 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:56:40.045603 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:56:40.047612 (+ 2009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:56:40.048037 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:56:40.048039 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:56:40.048039 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:56:40.048097 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:56:40.048348 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:56:40.048349 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:56:40.048349 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:56:40.048510 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:56:40.048569 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 21:56:40.048722 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 21:56:40.048797 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 21:56:40.048889 16281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:40.041317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.125:45213 (local address 127.0.0.1:41344)
0928 21:56:40.041356 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:40.041368 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:56:40.041402 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:56:40.041543 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:56:40.041543 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:56:40.041645 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:56:40.041679 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:56:40.041681 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:56:40.042471 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:56:40.042472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:56:40.043164 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:56:40.043166 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:56:40.043706 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:56:40.043707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:56:40.043752 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:40.044068 (+ 316us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:56:40.044072 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:56:40.045504 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:56:40.047690 (+ 2186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:56:40.047692 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:56:40.047692 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:56:40.047903 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:56:40.048190 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:56:40.048192 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:56:40.048192 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:56:40.048253 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:56:40.048572 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:56:40.048572 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:56:40.048623 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 21:56:40.048688 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 21:56:40.048787 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260928 21:56:40.049618 16279 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.182.124" port: 44351 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:56:40.049696 16132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:40.041424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59088 (local address 127.2.182.126:39923)
0928 21:56:40.041458 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 21:56:40.041460 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:56:40.041643 (+ 183us) server_negotiation.cc:407] Connection header received
0928 21:56:40.041680 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:56:40.041681 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:56:40.041690 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:56:40.041731 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:56:40.042284 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:40.042913 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:40.043854 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:40.044100 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:40.044171 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:40.046465 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:56:40.046466 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:56:40.046466 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:56:40.046473 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:56:40.047915 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:56:40.048450 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:56:40.048451 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:56:40.048451 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:56:40.048500 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:56:40.048673 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:56:40.048673 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:56:40.048674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:56:40.048805 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:56:40.049415 (+ 610us) server_negotiation.cc:1035] Waiting for connection context
0928 21:56:40.049450 (+ 35us) server_negotiation.cc:299] Negotiation successful
0928 21:56:40.049518 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
Sep 28 21:56:40 dist-test-slave-k6lz krb5kdc[16048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632600, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.124@KRBTEST.COM for kudu/127.2.182.124@KRBTEST.COM
Sep 28 21:56:40 dist-test-slave-k6lz krb5kdc[16048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.182.125@KRBTEST.COM
I20260928 21:56:40.055117 16129 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.182.124" port: 44351 } attempt: 2
I20260928 21:56:40.056694 16283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:40.050000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41870 (local address 127.2.182.124:44351)
0928 21:56:40.050050 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 21:56:40.050052 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:56:40.050105 (+ 53us) server_negotiation.cc:407] Connection header received
0928 21:56:40.050267 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:56:40.050268 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:56:40.050278 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:56:40.050322 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:56:40.050646 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:40.051161 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:40.051821 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:40.052023 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:40.052082 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:40.053744 (+ 1662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:56:40.053745 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:56:40.053745 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:56:40.053751 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:56:40.055556 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:56:40.055919 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:56:40.055920 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:56:40.055920 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:56:40.055971 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:56:40.056147 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:56:40.056148 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:56:40.056148 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:56:40.056272 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:56:40.056313 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 21:56:40.056493 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 21:56:40.056605 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":38}
I20260928 21:56:40.056685 16281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:40.049962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.124:44351 (local address 127.0.0.1:41870)
0928 21:56:40.050000 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:40.050065 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 21:56:40.050116 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:56:40.050372 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:56:40.050373 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:56:40.050517 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:56:40.050563 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:56:40.050564 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:56:40.051231 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:56:40.051232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:56:40.051755 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:56:40.051756 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:56:40.052085 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:56:40.052086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:56:40.052131 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:40.052495 (+ 364us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:56:40.052499 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:56:40.053656 (+ 1157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:56:40.055656 (+ 2000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:56:40.055657 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:56:40.055657 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:56:40.055858 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:56:40.056028 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:56:40.056029 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:56:40.056029 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:56:40.056084 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:56:40.056352 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:56:40.056353 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:56:40.056386 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 21:56:40.056464 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 21:56:40.056564 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260928 21:56:40.057579 16208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:40.050296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41354 (local address 127.2.182.125:45213)
0928 21:56:40.050338 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 21:56:40.050340 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:56:40.050391 (+ 51us) server_negotiation.cc:407] Connection header received
0928 21:56:40.050451 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:56:40.050452 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:56:40.050459 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:56:40.050492 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:56:40.050944 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:40.051445 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:40.052057 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:40.052249 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:40.052299 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:40.054321 (+ 2022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:56:40.054322 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:56:40.054323 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:56:40.054330 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:56:40.056005 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:56:40.056454 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:56:40.056455 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:56:40.056456 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:56:40.056521 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:56:40.056864 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:56:40.056865 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:56:40.056866 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:56:40.056981 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:56:40.057152 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 21:56:40.057395 (+ 243us) server_negotiation.cc:299] Negotiation successful
0928 21:56:40.057477 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 21:56:40.058395 16279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fe8733be2434a29ab1c7cd531d003a4: Bootstrap starting.
I20260928 21:56:40.059079 2777 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:56:40.059149 16279 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4fe8733be2434a29ab1c7cd531d003a4: Neither blocks nor log segments found. Creating new log.
I20260928 21:56:40.059520 16279 log.cc:824] T 00000000000000000000000000000000 P 4fe8733be2434a29ab1c7cd531d003a4: Log is configured to *not* fsync() on all Append() calls
Sep 28 21:56:40 dist-test-slave-k6lz krb5kdc[16048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632599, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.126@KRBTEST.COM for kudu/127.2.182.124@KRBTEST.COM
I20260928 21:56:40.060400 16279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fe8733be2434a29ab1c7cd531d003a4: No bootstrap required, opened a new log
I20260928 21:56:40.062467 16279 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4fe8733be2434a29ab1c7cd531d003a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74162e51f4584b91a89a86bf561a7e97" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 39923 } } peers { permanent_uuid: "d5c5939a108646dbaa66308d20b431f9" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 45213 } } peers { permanent_uuid: "4fe8733be2434a29ab1c7cd531d003a4" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 44351 } }
I20260928 21:56:40.062623 16279 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4fe8733be2434a29ab1c7cd531d003a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:56:40.062651 16279 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4fe8733be2434a29ab1c7cd531d003a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fe8733be2434a29ab1c7cd531d003a4, State: Initialized, Role: FOLLOWER
I20260928 21:56:40.062764 16279 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4fe8733be2434a29ab1c7cd531d003a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74162e51f4584b91a89a86bf561a7e97" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 39923 } } peers { permanent_uuid: "d5c5939a108646dbaa66308d20b431f9" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 45213 } } peers { permanent_uuid: "4fe8733be2434a29ab1c7cd531d003a4" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 44351 } }
I20260928 21:56:40.063042 16279 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4fe8733be2434a29ab1c7cd531d003a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:56:40.063311 16131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:40.055497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.124:44351 (local address 127.0.0.1:41886)
0928 21:56:40.055550 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:40.055559 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 21:56:40.055594 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:56:40.056001 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:56:40.056003 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:56:40.056137 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:56:40.056183 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:56:40.056185 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:56:40.057424 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:56:40.057426 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:56:40.058123 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:56:40.058125 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:56:40.058616 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:56:40.058617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:56:40.058677 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:40.059088 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:56:40.059093 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:56:40.060519 (+ 1426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:56:40.062255 (+ 1736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:56:40.062256 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:56:40.062257 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:56:40.062466 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:56:40.062650 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:56:40.062651 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:56:40.062651 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:56:40.062701 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:56:40.063024 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:56:40.063025 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:56:40.063068 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 21:56:40.063124 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 21:56:40.063214 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 21:56:40.063690 16284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:40.055671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41886 (local address 127.2.182.124:44351)
0928 21:56:40.055853 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 21:56:40.055855 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:56:40.055862 (+ 7us) server_negotiation.cc:407] Connection header received
0928 21:56:40.055881 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:56:40.055881 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:56:40.055892 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:56:40.056235 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:56:40.056270 (+ 35us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:40.057334 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:40.058225 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:40.058519 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:40.058609 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:40.060595 (+ 1986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:56:40.060597 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:56:40.060597 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:56:40.060604 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:56:40.062178 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:56:40.062540 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:56:40.062540 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:56:40.062541 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:56:40.062594 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:56:40.062778 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:56:40.062779 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:56:40.062780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:56:40.062941 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:56:40.062998 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 21:56:40.063503 (+ 505us) server_negotiation.cc:299] Negotiation successful
0928 21:56:40.063581 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":109,"threads_started":1}
I20260928 21:56:40.064414 16293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fe8733be2434a29ab1c7cd531d003a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74162e51f4584b91a89a86bf561a7e97" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 39923 } } peers { permanent_uuid: "d5c5939a108646dbaa66308d20b431f9" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 45213 } } peers { permanent_uuid: "4fe8733be2434a29ab1c7cd531d003a4" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 44351 } } }
I20260928 21:56:40.065171 16129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74162e51f4584b91a89a86bf561a7e97: Bootstrap starting.
I20260928 21:56:40.065722 16293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fe8733be2434a29ab1c7cd531d003a4 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:56:40.065902 16129 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74162e51f4584b91a89a86bf561a7e97: Neither blocks nor log segments found. Creating new log.
I20260928 21:56:40.066195 16129 log.cc:824] T 00000000000000000000000000000000 P 74162e51f4584b91a89a86bf561a7e97: Log is configured to *not* fsync() on all Append() calls
I20260928 21:56:40.068140 16283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:40.060659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41888 (local address 127.2.182.124:44351)
0928 21:56:40.060695 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 21:56:40.060697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:56:40.061177 (+ 480us) server_negotiation.cc:407] Connection header received
0928 21:56:40.061331 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:56:40.061332 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:56:40.061341 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:56:40.061368 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:56:40.062065 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:40.062745 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:40.063738 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:40.063998 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:40.064042 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:40.065497 (+ 1455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:56:40.065499 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:56:40.065499 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:56:40.065506 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:56:40.066883 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:56:40.067251 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:56:40.067252 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:56:40.067252 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:56:40.067317 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:56:40.067535 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:56:40.067536 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:56:40.067536 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:56:40.067654 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:56:40.067724 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 21:56:40.067989 (+ 265us) server_negotiation.cc:299] Negotiation successful
0928 21:56:40.068062 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 21:56:40.069204 16129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74162e51f4584b91a89a86bf561a7e97: No bootstrap required, opened a new log
I20260928 21:56:40.069422 16208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:40.060878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41360 (local address 127.2.182.125:45213)
0928 21:56:40.060916 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 21:56:40.060917 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:56:40.061171 (+ 254us) server_negotiation.cc:407] Connection header received
0928 21:56:40.061261 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:56:40.061262 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:56:40.061273 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:56:40.061299 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:56:40.062171 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:40.062835 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:40.063938 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:40.064200 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:40.064241 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:40.065983 (+ 1742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:56:40.065984 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:56:40.065984 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:56:40.065991 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:56:40.068128 (+ 2137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:56:40.068546 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:56:40.068548 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:56:40.068549 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:56:40.068607 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:56:40.068807 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:56:40.068808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:56:40.068808 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:56:40.068928 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:56:40.069016 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 21:56:40.069255 (+ 239us) server_negotiation.cc:299] Negotiation successful
0928 21:56:40.069333 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 21:56:40.068413 16211 master_runner.cc:186] Error getting master registration for 127.2.182.126:39923: OK, instance_id { permanent_uuid: "74162e51f4584b91a89a86bf561a7e97" instance_seqno: 1790632599712487 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 21:56:40.070433 16211 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 21:56:40.068421 16305 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4fe8733be2434a29ab1c7cd531d003a4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:56:40.070796 16305 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:56:40.071226 16129 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74162e51f4584b91a89a86bf561a7e97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74162e51f4584b91a89a86bf561a7e97" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 39923 } } peers { permanent_uuid: "d5c5939a108646dbaa66308d20b431f9" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 45213 } } peers { permanent_uuid: "4fe8733be2434a29ab1c7cd531d003a4" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 44351 } }
I20260928 21:56:40.071393 16129 raft_consensus.cc:374] T 00000000000000000000000000000000 P 74162e51f4584b91a89a86bf561a7e97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:56:40.071450 16129 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74162e51f4584b91a89a86bf561a7e97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74162e51f4584b91a89a86bf561a7e97, State: Initialized, Role: FOLLOWER
I20260928 21:56:40.071576 16129 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74162e51f4584b91a89a86bf561a7e97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74162e51f4584b91a89a86bf561a7e97" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 39923 } } peers { permanent_uuid: "d5c5939a108646dbaa66308d20b431f9" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 45213 } } peers { permanent_uuid: "4fe8733be2434a29ab1c7cd531d003a4" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 44351 } }
I20260928 21:56:40.071893 16129 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74162e51f4584b91a89a86bf561a7e97 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:56:40.071848 16307 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74162e51f4584b91a89a86bf561a7e97 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74162e51f4584b91a89a86bf561a7e97" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 39923 } } peers { permanent_uuid: "d5c5939a108646dbaa66308d20b431f9" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 45213 } } peers { permanent_uuid: "4fe8733be2434a29ab1c7cd531d003a4" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 44351 } } }
I20260928 21:56:40.072036 16307 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74162e51f4584b91a89a86bf561a7e97 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:56:40.072589 16132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:40.060484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59094 (local address 127.2.182.126:39923)
0928 21:56:40.060525 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 21:56:40.060527 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:56:40.061079 (+ 552us) server_negotiation.cc:407] Connection header received
0928 21:56:40.061432 (+ 353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:56:40.061433 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:56:40.061444 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:56:40.061521 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:56:40.064380 (+ 2859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:40.065060 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:40.065961 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:40.066236 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:40.066274 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:40.067556 (+ 1282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:56:40.067557 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:56:40.067557 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:56:40.067564 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:56:40.071411 (+ 3847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:56:40.071778 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:56:40.071779 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:56:40.071780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:56:40.071850 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:56:40.072051 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:56:40.072052 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:56:40.072053 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:56:40.072180 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:56:40.072249 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 21:56:40.072434 (+ 185us) server_negotiation.cc:299] Negotiation successful
0928 21:56:40.072510 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 21:56:40.073776 16318 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 74162e51f4584b91a89a86bf561a7e97: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:56:40.073873 16318 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:56:40.074815 16061 master_runner.cc:186] Error getting master registration for 127.2.182.125:45213: OK, instance_id { permanent_uuid: "d5c5939a108646dbaa66308d20b431f9" instance_seqno: 1790632599869906 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 21:56:40.076701 16061 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 21:56:40.103725 16204 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.182.124" port: 44351 } attempt: 3
Sep 28 21:56:40 dist-test-slave-k6lz krb5kdc[16048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632599, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.125@KRBTEST.COM for kudu/127.2.182.124@KRBTEST.COM
I20260928 21:56:40.110219 16283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:40.104361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41898 (local address 127.2.182.124:44351)
0928 21:56:40.104408 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 21:56:40.104410 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:56:40.104418 (+ 8us) server_negotiation.cc:407] Connection header received
0928 21:56:40.104439 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:56:40.104440 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:56:40.104450 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:56:40.104496 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:56:40.104835 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:40.105319 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:40.105972 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:40.106152 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:40.106213 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:40.107973 (+ 1760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:56:40.107975 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:56:40.107975 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:56:40.108006 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:56:40.109249 (+ 1243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:56:40.109542 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:56:40.109543 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:56:40.109544 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:56:40.109594 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:56:40.109751 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:56:40.109751 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:56:40.109752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:56:40.109863 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:56:40.109916 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 21:56:40.110048 (+ 132us) server_negotiation.cc:299] Negotiation successful
0928 21:56:40.110122 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 21:56:40.110229 16206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:40.104078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.124:44351 (local address 127.0.0.1:41898)
0928 21:56:40.104166 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:40.104253 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0928 21:56:40.104293 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:56:40.104518 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:56:40.104519 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:56:40.104710 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:56:40.104756 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:56:40.104758 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:56:40.105393 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:56:40.105394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:56:40.105895 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:56:40.105897 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:56:40.106210 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:56:40.106211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:56:40.106263 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:40.106670 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:56:40.106684 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:56:40.107891 (+ 1207us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:56:40.109325 (+ 1434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:56:40.109326 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:56:40.109327 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:56:40.109483 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:56:40.109645 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:56:40.109646 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:56:40.109646 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:56:40.109702 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:56:40.109914 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:56:40.109915 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:56:40.109962 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 21:56:40.110020 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 21:56:40.110122 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260928 21:56:40.111586 16204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5c5939a108646dbaa66308d20b431f9: Bootstrap starting.
I20260928 21:56:40.113358 16204 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5c5939a108646dbaa66308d20b431f9: Neither blocks nor log segments found. Creating new log.
I20260928 21:56:40.113662 16204 log.cc:824] T 00000000000000000000000000000000 P d5c5939a108646dbaa66308d20b431f9: Log is configured to *not* fsync() on all Append() calls
I20260928 21:56:40.114473 16204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5c5939a108646dbaa66308d20b431f9: No bootstrap required, opened a new log
I20260928 21:56:40.115975 16204 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5c5939a108646dbaa66308d20b431f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74162e51f4584b91a89a86bf561a7e97" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 39923 } } peers { permanent_uuid: "d5c5939a108646dbaa66308d20b431f9" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 45213 } } peers { permanent_uuid: "4fe8733be2434a29ab1c7cd531d003a4" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 44351 } }
I20260928 21:56:40.116091 16204 raft_consensus.cc:374] T 00000000000000000000000000000000 P d5c5939a108646dbaa66308d20b431f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:56:40.116123 16204 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5c5939a108646dbaa66308d20b431f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5c5939a108646dbaa66308d20b431f9, State: Initialized, Role: FOLLOWER
I20260928 21:56:40.116258 16204 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5c5939a108646dbaa66308d20b431f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74162e51f4584b91a89a86bf561a7e97" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 39923 } } peers { permanent_uuid: "d5c5939a108646dbaa66308d20b431f9" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 45213 } } peers { permanent_uuid: "4fe8733be2434a29ab1c7cd531d003a4" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 44351 } }
I20260928 21:56:40.116523 16321 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5c5939a108646dbaa66308d20b431f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74162e51f4584b91a89a86bf561a7e97" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 39923 } } peers { permanent_uuid: "d5c5939a108646dbaa66308d20b431f9" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 45213 } } peers { permanent_uuid: "4fe8733be2434a29ab1c7cd531d003a4" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 44351 } } }
I20260928 21:56:40.116585 16204 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5c5939a108646dbaa66308d20b431f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:56:40.116657 16321 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5c5939a108646dbaa66308d20b431f9 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 21:56:40.117766 16157 tablet.cc:2369] T 00000000000000000000000000000000 P d5c5939a108646dbaa66308d20b431f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 21:56:40.118482 16332 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d5c5939a108646dbaa66308d20b431f9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:56:40.118595 16332 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:56:40.120702 16136 master_runner.cc:263] No leader master found from master d5c5939a108646dbaa66308d20b431f9
I20260928 21:56:40.120759 16136 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 21:56:40.210433 16082 tablet.cc:2369] T 00000000000000000000000000000000 P 74162e51f4584b91a89a86bf561a7e97: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 21:56:40.274680 16232 tablet.cc:2369] T 00000000000000000000000000000000 P 4fe8733be2434a29ab1c7cd531d003a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:56:40.354593 16307 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74162e51f4584b91a89a86bf561a7e97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:56:40.354765 16307 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74162e51f4584b91a89a86bf561a7e97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74162e51f4584b91a89a86bf561a7e97" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 39923 } } peers { permanent_uuid: "d5c5939a108646dbaa66308d20b431f9" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 45213 } } peers { permanent_uuid: "4fe8733be2434a29ab1c7cd531d003a4" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 44351 } }
I20260928 21:56:40.355001 16307 leader_election.cc:288] T 00000000000000000000000000000000 P 74162e51f4584b91a89a86bf561a7e97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d5c5939a108646dbaa66308d20b431f9 (127.2.182.125:45213), 4fe8733be2434a29ab1c7cd531d003a4 (127.2.182.124:44351)
I20260928 21:56:40.355360 16253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "74162e51f4584b91a89a86bf561a7e97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fe8733be2434a29ab1c7cd531d003a4" is_pre_election: true
I20260928 21:56:40.355383 16178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "74162e51f4584b91a89a86bf561a7e97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5c5939a108646dbaa66308d20b431f9" is_pre_election: true
I20260928 21:56:40.355515 16253 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4fe8733be2434a29ab1c7cd531d003a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74162e51f4584b91a89a86bf561a7e97 in term 0.
I20260928 21:56:40.355520 16178 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d5c5939a108646dbaa66308d20b431f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74162e51f4584b91a89a86bf561a7e97 in term 0.
I20260928 21:56:40.355724 16077 leader_election.cc:302] T 00000000000000000000000000000000 P 74162e51f4584b91a89a86bf561a7e97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74162e51f4584b91a89a86bf561a7e97, d5c5939a108646dbaa66308d20b431f9; no voters:
I20260928 21:56:40.355867 16307 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74162e51f4584b91a89a86bf561a7e97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:56:40.355914 16307 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74162e51f4584b91a89a86bf561a7e97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:56:40.355934 16307 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74162e51f4584b91a89a86bf561a7e97 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:56:40.356638 16307 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74162e51f4584b91a89a86bf561a7e97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74162e51f4584b91a89a86bf561a7e97" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 39923 } } peers { permanent_uuid: "d5c5939a108646dbaa66308d20b431f9" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 45213 } } peers { permanent_uuid: "4fe8733be2434a29ab1c7cd531d003a4" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 44351 } }
I20260928 21:56:40.356782 16307 leader_election.cc:288] T 00000000000000000000000000000000 P 74162e51f4584b91a89a86bf561a7e97 [CANDIDATE]: Term 1 election: Requested vote from peers d5c5939a108646dbaa66308d20b431f9 (127.2.182.125:45213), 4fe8733be2434a29ab1c7cd531d003a4 (127.2.182.124:44351)
I20260928 21:56:40.356953 16178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "74162e51f4584b91a89a86bf561a7e97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5c5939a108646dbaa66308d20b431f9"
I20260928 21:56:40.357003 16253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "74162e51f4584b91a89a86bf561a7e97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fe8733be2434a29ab1c7cd531d003a4"
I20260928 21:56:40.357048 16178 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5c5939a108646dbaa66308d20b431f9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:56:40.357071 16253 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4fe8733be2434a29ab1c7cd531d003a4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:56:40.357738 16253 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4fe8733be2434a29ab1c7cd531d003a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74162e51f4584b91a89a86bf561a7e97 in term 1.
I20260928 21:56:40.357856 16178 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d5c5939a108646dbaa66308d20b431f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74162e51f4584b91a89a86bf561a7e97 in term 1.
I20260928 21:56:40.357908 16079 leader_election.cc:302] T 00000000000000000000000000000000 P 74162e51f4584b91a89a86bf561a7e97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fe8733be2434a29ab1c7cd531d003a4, 74162e51f4584b91a89a86bf561a7e97; no voters:
I20260928 21:56:40.357990 16307 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74162e51f4584b91a89a86bf561a7e97 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:56:40.358115 16307 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74162e51f4584b91a89a86bf561a7e97 [term 1 LEADER]: Becoming Leader. State: Replica: 74162e51f4584b91a89a86bf561a7e97, State: Running, Role: LEADER
I20260928 21:56:40.358209 16307 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74162e51f4584b91a89a86bf561a7e97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74162e51f4584b91a89a86bf561a7e97" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 39923 } } peers { permanent_uuid: "d5c5939a108646dbaa66308d20b431f9" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 45213 } } peers { permanent_uuid: "4fe8733be2434a29ab1c7cd531d003a4" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 44351 } }
I20260928 21:56:40.358592 16334 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74162e51f4584b91a89a86bf561a7e97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74162e51f4584b91a89a86bf561a7e97. Latest consensus state: current_term: 1 leader_uuid: "74162e51f4584b91a89a86bf561a7e97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74162e51f4584b91a89a86bf561a7e97" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 39923 } } peers { permanent_uuid: "d5c5939a108646dbaa66308d20b431f9" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 45213 } } peers { permanent_uuid: "4fe8733be2434a29ab1c7cd531d003a4" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 44351 } } }
I20260928 21:56:40.358685 16334 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74162e51f4584b91a89a86bf561a7e97 [sys.catalog]: This master's current role is: LEADER
I20260928 21:56:40.358881 16336 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:56:40.359057 16336 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:56:40.360572 16253 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4fe8733be2434a29ab1c7cd531d003a4 [term 1 FOLLOWER]: Refusing update from remote peer 74162e51f4584b91a89a86bf561a7e97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:56:40.361346 16178 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d5c5939a108646dbaa66308d20b431f9 [term 1 FOLLOWER]: Refusing update from remote peer 74162e51f4584b91a89a86bf561a7e97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:56:40.361778 16307 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 74162e51f4584b91a89a86bf561a7e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fe8733be2434a29ab1c7cd531d003a4" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 44351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:56:40.361966 16334 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 74162e51f4584b91a89a86bf561a7e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5c5939a108646dbaa66308d20b431f9" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 45213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:56:40.363615 16293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fe8733be2434a29ab1c7cd531d003a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74162e51f4584b91a89a86bf561a7e97. Latest consensus state: current_term: 1 leader_uuid: "74162e51f4584b91a89a86bf561a7e97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74162e51f4584b91a89a86bf561a7e97" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 39923 } } peers { permanent_uuid: "d5c5939a108646dbaa66308d20b431f9" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 45213 } } peers { permanent_uuid: "4fe8733be2434a29ab1c7cd531d003a4" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 44351 } } }
I20260928 21:56:40.363732 16293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fe8733be2434a29ab1c7cd531d003a4 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:56:40.364884 16321 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5c5939a108646dbaa66308d20b431f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74162e51f4584b91a89a86bf561a7e97. Latest consensus state: current_term: 1 leader_uuid: "74162e51f4584b91a89a86bf561a7e97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74162e51f4584b91a89a86bf561a7e97" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 39923 } } peers { permanent_uuid: "d5c5939a108646dbaa66308d20b431f9" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 45213 } } peers { permanent_uuid: "4fe8733be2434a29ab1c7cd531d003a4" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 44351 } } }
I20260928 21:56:40.364982 16321 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5c5939a108646dbaa66308d20b431f9 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:56:40.365448 16307 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74162e51f4584b91a89a86bf561a7e97 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "74162e51f4584b91a89a86bf561a7e97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74162e51f4584b91a89a86bf561a7e97" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 39923 } } peers { permanent_uuid: "d5c5939a108646dbaa66308d20b431f9" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 45213 } } peers { permanent_uuid: "4fe8733be2434a29ab1c7cd531d003a4" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 44351 } } }
I20260928 21:56:40.365537 16307 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74162e51f4584b91a89a86bf561a7e97 [sys.catalog]: This master's current role is: LEADER
I20260928 21:56:40.365582 16337 mvcc.cc:205] Tried to move back new op lower bound from 7334431131074596864 to 7334431131067813888. Current Snapshot: MvccSnapshot[applied={T|T < 7334431131074596864}]
I20260928 21:56:40.365829 16336 catalog_manager.cc:1347] Generated new cluster ID: 1f11f52cb01c4329b181baf92e8bee0c
I20260928 21:56:40.365880 16336 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:56:40.366026 16293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fe8733be2434a29ab1c7cd531d003a4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "74162e51f4584b91a89a86bf561a7e97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74162e51f4584b91a89a86bf561a7e97" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 39923 } } peers { permanent_uuid: "d5c5939a108646dbaa66308d20b431f9" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 45213 } } peers { permanent_uuid: "4fe8733be2434a29ab1c7cd531d003a4" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 44351 } } }
I20260928 21:56:40.366102 16293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fe8733be2434a29ab1c7cd531d003a4 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:56:40.366889 16307 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74162e51f4584b91a89a86bf561a7e97 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "74162e51f4584b91a89a86bf561a7e97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74162e51f4584b91a89a86bf561a7e97" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 39923 } } peers { permanent_uuid: "d5c5939a108646dbaa66308d20b431f9" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 45213 } } peers { permanent_uuid: "4fe8733be2434a29ab1c7cd531d003a4" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 44351 } } }
I20260928 21:56:40.366972 16307 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74162e51f4584b91a89a86bf561a7e97 [sys.catalog]: This master's current role is: LEADER
I20260928 21:56:40.367558 16321 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5c5939a108646dbaa66308d20b431f9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "74162e51f4584b91a89a86bf561a7e97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74162e51f4584b91a89a86bf561a7e97" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 39923 } } peers { permanent_uuid: "d5c5939a108646dbaa66308d20b431f9" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 45213 } } peers { permanent_uuid: "4fe8733be2434a29ab1c7cd531d003a4" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 44351 } } }
I20260928 21:56:40.367642 16321 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5c5939a108646dbaa66308d20b431f9 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:56:40.377406 16336 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:56:40.377964 16336 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:56:40.383154 16336 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 74162e51f4584b91a89a86bf561a7e97: Generated new TSK 0
I20260928 21:56:40.383399 16336 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:56:41.071301 16305 catalog_manager.cc:1259] Loaded cluster ID: 1f11f52cb01c4329b181baf92e8bee0c
I20260928 21:56:41.071417 16305 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4fe8733be2434a29ab1c7cd531d003a4: loading cluster ID for follower catalog manager: success
I20260928 21:56:41.072322 16305 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4fe8733be2434a29ab1c7cd531d003a4: acquiring CA information for follower catalog manager: success
I20260928 21:56:41.072697 16305 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4fe8733be2434a29ab1c7cd531d003a4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 21:56:41.119266 16332 catalog_manager.cc:1259] Loaded cluster ID: 1f11f52cb01c4329b181baf92e8bee0c
I20260928 21:56:41.119568 16332 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d5c5939a108646dbaa66308d20b431f9: loading cluster ID for follower catalog manager: success
I20260928 21:56:41.120699 16332 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d5c5939a108646dbaa66308d20b431f9: acquiring CA information for follower catalog manager: success
I20260928 21:56:41.121039 16332 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d5c5939a108646dbaa66308d20b431f9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 21:56:41.433817 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 16061
I20260928 21:56:41.438705 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 16136
I20260928 21:56:41.443358 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 16211
2026-09-28T21:56:41Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2934 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632440055392-2777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:56:41 dist-test-slave-k6lz krb5kdc[16351](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:56:41 dist-test-slave-k6lz krb5kdc[16351](info): set up 2 sockets
Sep 28 21:56:41 dist-test-slave-k6lz krb5kdc[16351](info): commencing operation
krb5kdc: starting...
W20260928 21:56:42.407138 2777 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Sep 28 21:56:42 dist-test-slave-k6lz krb5kdc[16351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632602, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:56:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:56:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab.
Entry for principal kudu/127.2.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab.
I20260928 21:56:42.539296 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632440055392-2777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632440055392-2777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632440055392-2777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:34659
--webserver_interface=127.2.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab
--principal=kudu/127.2.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:35151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.126:34659
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:56:42.625614 16364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:56:42.625945 16364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:56:42.626017 16364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:56:42.627455 16364 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:56:42.627511 16364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:56:42.627532 16364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:56:42.627547 16364 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:56:42.627563 16364 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:56:42.629261 16364 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:35151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632440055392-2777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632440055392-2777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.126:34659
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab
--principal=kudu/127.2.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:34659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16364
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632440055392-2777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:56:42.629535 16364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:56:42.629763 16364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:56:42.634919 16372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:56:42.634974 16364 server_base.cc:1034] running on GCE node
W20260928 21:56:42.634941 16370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:56:42.634936 16369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:56:42.635484 16364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:56:42.635778 16364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:56:42.636948 16364 hybrid_clock.cc:648] HybridClock initialized: now 1790632602636924 us; error 38 us; skew 500 ppm
Sep 28 21:56:42 dist-test-slave-k6lz krb5kdc[16351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632602, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:56:42.639611 16364 init.cc:377] Logged in from keytab as kudu/127.2.182.126@KRBTEST.COM (short username kudu)
I20260928 21:56:42.640095 16364 webserver.cc:460] Webserver started at http://127.2.182.126:43769/ using document root <none> and password file <none>
I20260928 21:56:42.640328 16364 fs_manager.cc:359] Metadata directory not provided
I20260928 21:56:42.640373 16364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:56:42.640494 16364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:56:42.641561 16364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632440055392-2777-0/minicluster-data/master-0/data/instance:
uuid: "dd99e482f8694898810397f985e9de6f"
format_stamp: "Formatted at 2026-09-28 21:56:42 on dist-test-slave-k6lz"
server_key: "be3c58c4e70dc40c4590e97479d368f3"
server_key_iv: "9afed19243e1ec7e04f4b832ebbd2c53"
server_key_version: "encryptionkey@0"
I20260928 21:56:42.641870 16364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632440055392-2777-0/minicluster-data/master-0/wal/instance:
uuid: "dd99e482f8694898810397f985e9de6f"
format_stamp: "Formatted at 2026-09-28 21:56:42 on dist-test-slave-k6lz"
server_key: "be3c58c4e70dc40c4590e97479d368f3"
server_key_iv: "9afed19243e1ec7e04f4b832ebbd2c53"
server_key_version: "encryptionkey@0"
I20260928 21:56:42.643254 16364 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:56:42.644088 16379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:56:42.644336 16364 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:56:42.644429 16364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632440055392-2777-0/minicluster-data/master-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632440055392-2777-0/minicluster-data/master-0/wal
uuid: "dd99e482f8694898810397f985e9de6f"
format_stamp: "Formatted at 2026-09-28 21:56:42 on dist-test-slave-k6lz"
server_key: "be3c58c4e70dc40c4590e97479d368f3"
server_key_iv: "9afed19243e1ec7e04f4b832ebbd2c53"
server_key_version: "encryptionkey@0"
I20260928 21:56:42.644491 16364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632440055392-2777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632440055392-2777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632440055392-2777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:56:42.680119 16364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:56:42.682615 16364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:56:42.682763 16364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:56:42.687469 16364 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.126:34659
I20260928 21:56:42.687513 16431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.126:34659 every 8 connection(s)
I20260928 21:56:42.687850 16364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
I20260928 21:56:42.688840 16432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:56:42.691155 16432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd99e482f8694898810397f985e9de6f: Bootstrap starting.
I20260928 21:56:42.691785 16432 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd99e482f8694898810397f985e9de6f: Neither blocks nor log segments found. Creating new log.
I20260928 21:56:42.692021 16432 log.cc:824] T 00000000000000000000000000000000 P dd99e482f8694898810397f985e9de6f: Log is configured to *not* fsync() on all Append() calls
I20260928 21:56:42.692778 16432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd99e482f8694898810397f985e9de6f: No bootstrap required, opened a new log
I20260928 21:56:42.694536 16432 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd99e482f8694898810397f985e9de6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd99e482f8694898810397f985e9de6f" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 34659 } }
I20260928 21:56:42.694694 16432 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd99e482f8694898810397f985e9de6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:56:42.694717 16432 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd99e482f8694898810397f985e9de6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd99e482f8694898810397f985e9de6f, State: Initialized, Role: FOLLOWER
I20260928 21:56:42.694823 16432 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd99e482f8694898810397f985e9de6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd99e482f8694898810397f985e9de6f" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 34659 } }
I20260928 21:56:42.694881 16432 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd99e482f8694898810397f985e9de6f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:56:42.694908 16432 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd99e482f8694898810397f985e9de6f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:56:42.694950 16432 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd99e482f8694898810397f985e9de6f [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:56:42.695384 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 16364
I20260928 21:56:42.695477 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632440055392-2777-0/minicluster-data/master-0/wal/instance
I20260928 21:56:42.695667 2777 external_mini_cluster.cc:1428] Setting key 941672eecd27ee266fbac35e53f942d9
I20260928 21:56:42.695706 16432 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd99e482f8694898810397f985e9de6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd99e482f8694898810397f985e9de6f" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 34659 } }
I20260928 21:56:42.695815 16432 leader_election.cc:302] T 00000000000000000000000000000000 P dd99e482f8694898810397f985e9de6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd99e482f8694898810397f985e9de6f; no voters:
I20260928 21:56:42.696002 16432 leader_election.cc:288] T 00000000000000000000000000000000 P dd99e482f8694898810397f985e9de6f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:56:42.696053 16435 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd99e482f8694898810397f985e9de6f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:56:42.696283 16435 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd99e482f8694898810397f985e9de6f [term 1 LEADER]: Becoming Leader. State: Replica: dd99e482f8694898810397f985e9de6f, State: Running, Role: LEADER
I20260928 21:56:42.696322 16432 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd99e482f8694898810397f985e9de6f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:56:42.696473 16435 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd99e482f8694898810397f985e9de6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd99e482f8694898810397f985e9de6f" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 34659 } }
I20260928 21:56:42.696861 16437 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd99e482f8694898810397f985e9de6f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd99e482f8694898810397f985e9de6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd99e482f8694898810397f985e9de6f" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 34659 } } }
I20260928 21:56:42.696875 16439 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd99e482f8694898810397f985e9de6f [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd99e482f8694898810397f985e9de6f. Latest consensus state: current_term: 1 leader_uuid: "dd99e482f8694898810397f985e9de6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd99e482f8694898810397f985e9de6f" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 34659 } } }
I20260928 21:56:42.697014 16437 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd99e482f8694898810397f985e9de6f [sys.catalog]: This master's current role is: LEADER
I20260928 21:56:42.697058 16439 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd99e482f8694898810397f985e9de6f [sys.catalog]: This master's current role is: LEADER
I20260928 21:56:42.697445 16444 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:56:42.697970 16444 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:56:42.699684 16444 catalog_manager.cc:1347] Generated new cluster ID: c78253e2685746c5b7419b385a051e60
I20260928 21:56:42.699740 16444 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 21:56:42 dist-test-slave-k6lz krb5kdc[16351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632602, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:56:42.709961 16438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:42.696161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46374 (local address 127.2.182.126:34659)
0928 21:56:42.696431 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0928 21:56:42.696442 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 21:56:42.698129 (+ 1687us) server_negotiation.cc:407] Connection header received
0928 21:56:42.698721 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:56:42.698731 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:56:42.698800 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:56:42.698951 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:56:42.699736 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:42.701129 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:42.702216 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:42.702869 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:42.702964 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:42.706134 (+ 3170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:56:42.706140 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:56:42.706140 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:56:42.706156 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:56:42.708122 (+ 1966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:56:42.708525 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:56:42.708528 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:56:42.708528 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:56:42.708647 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:56:42.708853 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:56:42.708854 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:56:42.708854 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:56:42.709196 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:56:42.709360 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 21:56:42.709428 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 21:56:42.709699 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":209,"thread_start_us":87,"threads_started":1}
I20260928 21:56:42.711299 2777 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:56:42.714040 16444 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:56:42.714818 16444 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:56:42.719100 16444 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd99e482f8694898810397f985e9de6f: Generated new TSK 0
I20260928 21:56:42.719359 16444 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:56:42.765506 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 16364
2026-09-28T21:56:42Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1320 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:56:42 dist-test-slave-k6lz krb5kdc[16459](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:56:42 dist-test-slave-k6lz krb5kdc[16459](info): set up 2 sockets
Sep 28 21:56:42 dist-test-slave-k6lz krb5kdc[16459](info): commencing operation
krb5kdc: starting...
W20260928 21:56:43.723433 2777 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Sep 28 21:56:43 dist-test-slave-k6lz krb5kdc[16459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:56:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:56:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab.
Entry for principal kudu/127.2.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab.
I20260928 21:56:43.847537 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:46141
--webserver_interface=127.2.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab
--principal=kudu/127.2.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:33315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.126:46141
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:56:43.930380 16472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:56:43.930619 16472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:56:43.930641 16472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:56:43.931861 16472 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:56:43.931903 16472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:56:43.931916 16472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:56:43.931927 16472 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:56:43.931939 16472 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:56:43.933524 16472 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:33315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.126:46141
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab
--principal=kudu/127.2.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:46141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:56:43.933779 16472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:56:43.934005 16472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:56:43.939160 16472 server_base.cc:1034] running on GCE node
W20260928 21:56:43.939141 16477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:56:43.939255 16478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:56:43.939117 16480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:56:43.939690 16472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:56:43.940050 16472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:56:43.943305 16472 hybrid_clock.cc:648] HybridClock initialized: now 1790632603943267 us; error 45 us; skew 500 ppm
Sep 28 21:56:43 dist-test-slave-k6lz krb5kdc[16459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632603, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:56:43.946614 16472 init.cc:377] Logged in from keytab as kudu/127.2.182.126@KRBTEST.COM (short username kudu)
I20260928 21:56:43.947119 16472 webserver.cc:460] Webserver started at http://127.2.182.126:42153/ using document root <none> and password file <none>
I20260928 21:56:43.947368 16472 fs_manager.cc:359] Metadata directory not provided
I20260928 21:56:43.947430 16472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:56:43.947559 16472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:56:43.948616 16472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/master-0/data/instance:
uuid: "73350ea0b7a94a888be60798c3d78cb2"
format_stamp: "Formatted at 2026-09-28 21:56:43 on dist-test-slave-k6lz"
server_key: "00124e733a592de95d6bf3109f613237"
server_key_iv: "5689f5249293130e727d1f0feb04321f"
server_key_version: "encryptionkey@0"
I20260928 21:56:43.948956 16472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/master-0/wal/instance:
uuid: "73350ea0b7a94a888be60798c3d78cb2"
format_stamp: "Formatted at 2026-09-28 21:56:43 on dist-test-slave-k6lz"
server_key: "00124e733a592de95d6bf3109f613237"
server_key_iv: "5689f5249293130e727d1f0feb04321f"
server_key_version: "encryptionkey@0"
I20260928 21:56:43.950404 16472 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 21:56:43.951272 16487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:56:43.951488 16472 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:56:43.951586 16472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/master-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/master-0/wal
uuid: "73350ea0b7a94a888be60798c3d78cb2"
format_stamp: "Formatted at 2026-09-28 21:56:43 on dist-test-slave-k6lz"
server_key: "00124e733a592de95d6bf3109f613237"
server_key_iv: "5689f5249293130e727d1f0feb04321f"
server_key_version: "encryptionkey@0"
I20260928 21:56:43.951687 16472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:56:43.960610 16472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:56:43.962641 16472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:56:43.962796 16472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:56:43.967130 16472 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.126:46141
I20260928 21:56:43.967235 16539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.126:46141 every 8 connection(s)
I20260928 21:56:43.967530 16472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
I20260928 21:56:43.968655 16540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:56:43.970991 16540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73350ea0b7a94a888be60798c3d78cb2: Bootstrap starting.
I20260928 21:56:43.971618 16540 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73350ea0b7a94a888be60798c3d78cb2: Neither blocks nor log segments found. Creating new log.
I20260928 21:56:43.971899 16540 log.cc:824] T 00000000000000000000000000000000 P 73350ea0b7a94a888be60798c3d78cb2: Log is configured to *not* fsync() on all Append() calls
I20260928 21:56:43.972730 16540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73350ea0b7a94a888be60798c3d78cb2: No bootstrap required, opened a new log
I20260928 21:56:43.974538 16540 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73350ea0b7a94a888be60798c3d78cb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73350ea0b7a94a888be60798c3d78cb2" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 46141 } }
I20260928 21:56:43.974763 16540 raft_consensus.cc:374] T 00000000000000000000000000000000 P 73350ea0b7a94a888be60798c3d78cb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:56:43.974792 16540 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73350ea0b7a94a888be60798c3d78cb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73350ea0b7a94a888be60798c3d78cb2, State: Initialized, Role: FOLLOWER
I20260928 21:56:43.974939 16540 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73350ea0b7a94a888be60798c3d78cb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73350ea0b7a94a888be60798c3d78cb2" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 46141 } }
I20260928 21:56:43.975008 16540 raft_consensus.cc:388] T 00000000000000000000000000000000 P 73350ea0b7a94a888be60798c3d78cb2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:56:43.975028 16540 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73350ea0b7a94a888be60798c3d78cb2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:56:43.975059 16540 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73350ea0b7a94a888be60798c3d78cb2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:56:43.975422 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 16472
I20260928 21:56:43.975523 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/master-0/wal/instance
I20260928 21:56:43.975728 2777 external_mini_cluster.cc:1428] Setting key 2a386459107307c37741d93ab54b181d
I20260928 21:56:43.976351 16540 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73350ea0b7a94a888be60798c3d78cb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73350ea0b7a94a888be60798c3d78cb2" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 46141 } }
I20260928 21:56:43.976478 16540 leader_election.cc:302] T 00000000000000000000000000000000 P 73350ea0b7a94a888be60798c3d78cb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73350ea0b7a94a888be60798c3d78cb2; no voters:
I20260928 21:56:43.976661 16540 leader_election.cc:288] T 00000000000000000000000000000000 P 73350ea0b7a94a888be60798c3d78cb2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:56:43.976683 16545 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73350ea0b7a94a888be60798c3d78cb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:56:43.976791 16545 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73350ea0b7a94a888be60798c3d78cb2 [term 1 LEADER]: Becoming Leader. State: Replica: 73350ea0b7a94a888be60798c3d78cb2, State: Running, Role: LEADER
I20260928 21:56:43.976936 16540 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73350ea0b7a94a888be60798c3d78cb2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:56:43.976927 16545 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73350ea0b7a94a888be60798c3d78cb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73350ea0b7a94a888be60798c3d78cb2" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 46141 } }
I20260928 21:56:43.977407 16546 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73350ea0b7a94a888be60798c3d78cb2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73350ea0b7a94a888be60798c3d78cb2. Latest consensus state: current_term: 1 leader_uuid: "73350ea0b7a94a888be60798c3d78cb2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73350ea0b7a94a888be60798c3d78cb2" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 46141 } } }
I20260928 21:56:43.977494 16546 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73350ea0b7a94a888be60798c3d78cb2 [sys.catalog]: This master's current role is: LEADER
I20260928 21:56:43.977689 16547 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73350ea0b7a94a888be60798c3d78cb2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "73350ea0b7a94a888be60798c3d78cb2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73350ea0b7a94a888be60798c3d78cb2" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 46141 } } }
I20260928 21:56:43.977749 16547 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73350ea0b7a94a888be60798c3d78cb2 [sys.catalog]: This master's current role is: LEADER
I20260928 21:56:43.979427 16549 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:56:43.980134 16549 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:56:43.982697 16549 catalog_manager.cc:1347] Generated new cluster ID: 71eab75250cc46ec98c3702a895ec83d
I20260928 21:56:43.982750 16549 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 21:56:43 dist-test-slave-k6lz krb5kdc[16459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:56:43.987771 16544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:43.976362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60916 (local address 127.2.182.126:46141)
0928 21:56:43.976579 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 21:56:43.976590 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 21:56:43.976623 (+ 33us) server_negotiation.cc:407] Connection header received
0928 21:56:43.976736 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:56:43.976743 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:56:43.976805 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:56:43.977839 (+ 1034us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:56:43.977941 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:43.979083 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:43.980267 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:43.980942 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:43.981083 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:43.983813 (+ 2730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:56:43.983817 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:56:43.983818 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:56:43.983831 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:56:43.985956 (+ 2125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:56:43.986371 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:56:43.986375 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:56:43.986376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:56:43.986525 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:56:43.986758 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:56:43.986759 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:56:43.986760 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:56:43.987110 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:56:43.987265 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 21:56:43.987391 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 21:56:43.987547 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":84,"threads_started":1}
I20260928 21:56:43.993373 16549 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:56:43.993963 16549 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:56:44.002015 16549 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 73350ea0b7a94a888be60798c3d78cb2: Generated new TSK 0
I20260928 21:56:44.002372 16549 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.182.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab.
Entry for principal kudu/127.2.182.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab.
I20260928 21:56:44.039134 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.65:0
--local_ip_for_outbound_sockets=127.2.182.65
--webserver_interface=127.2.182.65
--webserver_port=0
--tserver_master_addrs=127.2.182.126:46141
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab
--principal=kudu/127.2.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:33315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:56:44.126125 16566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:56:44.126338 16566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:56:44.126362 16566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:56:44.127566 16566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:56:44.127610 16566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:56:44.127641 16566 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.65
I20260928 21:56:44.129109 16566 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:33315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab
--principal=kudu/127.2.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.182.65
--webserver_port=0
--tserver_master_addrs=127.2.182.126:46141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16566
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.65
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:56:44.129303 16566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:56:44.129488 16566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:56:44.134562 16572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:56:44.134641 16566 server_base.cc:1034] running on GCE node
W20260928 21:56:44.134667 16571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:56:44.134562 16574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:56:44.134932 16566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:56:44.135133 16566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:56:44.136282 16566 hybrid_clock.cc:648] HybridClock initialized: now 1790632604136257 us; error 43 us; skew 500 ppm
Sep 28 21:56:44 dist-test-slave-k6lz krb5kdc[16459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632604, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:56:44.138942 16566 init.cc:377] Logged in from keytab as kudu/127.2.182.65@KRBTEST.COM (short username kudu)
I20260928 21:56:44.139443 16566 webserver.cc:460] Webserver started at http://127.2.182.65:37315/ using document root <none> and password file <none>
I20260928 21:56:44.139652 16566 fs_manager.cc:359] Metadata directory not provided
I20260928 21:56:44.139699 16566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:56:44.139817 16566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:56:44.140856 16566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-0/data/instance:
uuid: "f91d4c3aa32644a28c325bfc0c55b068"
format_stamp: "Formatted at 2026-09-28 21:56:44 on dist-test-slave-k6lz"
server_key: "10b3104d0bdea2c40b77e84c40c1f9e1"
server_key_iv: "3595306e0995fcca1342728259a51ad5"
server_key_version: "encryptionkey@0"
I20260928 21:56:44.141177 16566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-0/wal/instance:
uuid: "f91d4c3aa32644a28c325bfc0c55b068"
format_stamp: "Formatted at 2026-09-28 21:56:44 on dist-test-slave-k6lz"
server_key: "10b3104d0bdea2c40b77e84c40c1f9e1"
server_key_iv: "3595306e0995fcca1342728259a51ad5"
server_key_version: "encryptionkey@0"
I20260928 21:56:44.142481 16566 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 21:56:44.143253 16581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:56:44.143460 16566 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:56:44.143553 16566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-0/wal
uuid: "f91d4c3aa32644a28c325bfc0c55b068"
format_stamp: "Formatted at 2026-09-28 21:56:44 on dist-test-slave-k6lz"
server_key: "10b3104d0bdea2c40b77e84c40c1f9e1"
server_key_iv: "3595306e0995fcca1342728259a51ad5"
server_key_version: "encryptionkey@0"
I20260928 21:56:44.143640 16566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:56:44.164989 16566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:56:44.167277 16566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:56:44.167472 16566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:56:44.167721 16566 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:56:44.168027 16566 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:56:44.168061 16566 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:56:44.168089 16566 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:56:44.168107 16566 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:56:44.173615 16566 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.65:35749
I20260928 21:56:44.173684 16694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.65:35749 every 8 connection(s)
I20260928 21:56:44.173987 16566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
I20260928 21:56:44.175047 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 16566
I20260928 21:56:44.175143 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-0/wal/instance
I20260928 21:56:44.175408 2777 external_mini_cluster.cc:1428] Setting key 3a993a6721f488ee215dc2666aebd3cb
Sep 28 21:56:44 dist-test-slave-k6lz krb5kdc[16459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632604, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.65@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.182.66@KRBTEST.COM; defaulting to no policy
I20260928 21:56:44.184593 16544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:44.174790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.65:51165 (local address 127.2.182.126:46141)
0928 21:56:44.174857 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 21:56:44.174859 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:56:44.175063 (+ 204us) server_negotiation.cc:407] Connection header received
0928 21:56:44.175662 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:56:44.175663 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:56:44.175682 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:56:44.175742 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:56:44.176724 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:44.177416 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:44.178758 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:44.179041 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:44.179114 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:44.181685 (+ 2571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:56:44.181687 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:56:44.181687 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:56:44.181698 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:56:44.183160 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:56:44.183609 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:56:44.183610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:56:44.183611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:56:44.183674 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:56:44.183904 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:56:44.183905 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:56:44.183905 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:56:44.184034 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:56:44.184088 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 21:56:44.184386 (+ 298us) server_negotiation.cc:299] Negotiation successful
0928 21:56:44.184495 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 21:56:44.184741 16697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:44.174745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:46141 (local address 127.2.182.65:51165)
0928 21:56:44.174929 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:44.174973 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 21:56:44.175561 (+ 588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:56:44.175825 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:56:44.175828 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:56:44.176419 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:56:44.176619 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:56:44.176624 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:56:44.177571 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:56:44.177572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:56:44.178640 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:56:44.178644 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:56:44.179129 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:56:44.179130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:56:44.179257 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:44.179912 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:56:44.179921 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:56:44.181580 (+ 1659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:56:44.183265 (+ 1685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:56:44.183273 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:56:44.183275 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:56:44.183542 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:56:44.183733 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:56:44.183734 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:56:44.183735 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:56:44.183842 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:56:44.184089 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:56:44.184096 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:56:44.184255 (+ 159us) client_negotiation.cc:769] Sending connection context
0928 21:56:44.184344 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 21:56:44.184513 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":84,"threads_started":1}
I20260928 21:56:44.185320 16695 heartbeater.cc:347] Connected to a master server at 127.2.182.126:46141
I20260928 21:56:44.185420 16695 heartbeater.cc:464] Registering TS with master...
I20260928 21:56:44.185601 16695 heartbeater.cc:511] Master 127.2.182.126:46141 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.66@KRBTEST.COM" created.
I20260928 21:56:44.186183 16504 ts_manager.cc:194] Registered new tserver with Master: f91d4c3aa32644a28c325bfc0c55b068 (127.2.182.65:35749)
I20260928 21:56:44.186818 16504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.65@KRBTEST.COM'} at 127.2.182.65:51165
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab.
Entry for principal kudu/127.2.182.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab.
I20260928 21:56:44.204066 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.66:0
--local_ip_for_outbound_sockets=127.2.182.66
--webserver_interface=127.2.182.66
--webserver_port=0
--tserver_master_addrs=127.2.182.126:46141
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab
--principal=kudu/127.2.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:33315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:56:44.286943 16700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:56:44.287184 16700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:56:44.287211 16700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:56:44.288394 16700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:56:44.288430 16700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:56:44.288456 16700 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.66
I20260928 21:56:44.289918 16700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:33315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab
--principal=kudu/127.2.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.182.66
--webserver_port=0
--tserver_master_addrs=127.2.182.126:46141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16700
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.66
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:56:44.290107 16700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:56:44.290294 16700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:56:44.295158 16705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:56:44.295145 16708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:56:44.295145 16706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:56:44.295401 16700 server_base.cc:1034] running on GCE node
I20260928 21:56:44.295596 16700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:56:44.295856 16700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:56:44.297008 16700 hybrid_clock.cc:648] HybridClock initialized: now 1790632604296995 us; error 26 us; skew 500 ppm
Sep 28 21:56:44 dist-test-slave-k6lz krb5kdc[16459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632604, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:56:44.299455 16700 init.cc:377] Logged in from keytab as kudu/127.2.182.66@KRBTEST.COM (short username kudu)
I20260928 21:56:44.299885 16700 webserver.cc:460] Webserver started at http://127.2.182.66:45681/ using document root <none> and password file <none>
I20260928 21:56:44.300096 16700 fs_manager.cc:359] Metadata directory not provided
I20260928 21:56:44.300137 16700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:56:44.300259 16700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:56:44.301304 16700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-1/data/instance:
uuid: "69a6ebbd4f834658854802f999deed03"
format_stamp: "Formatted at 2026-09-28 21:56:44 on dist-test-slave-k6lz"
server_key: "e7d421bfce3f3143d1eb912196f2d41d"
server_key_iv: "711d2ac6a7ace06e7156feee47c58646"
server_key_version: "encryptionkey@0"
I20260928 21:56:44.301651 16700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-1/wal/instance:
uuid: "69a6ebbd4f834658854802f999deed03"
format_stamp: "Formatted at 2026-09-28 21:56:44 on dist-test-slave-k6lz"
server_key: "e7d421bfce3f3143d1eb912196f2d41d"
server_key_iv: "711d2ac6a7ace06e7156feee47c58646"
server_key_version: "encryptionkey@0"
I20260928 21:56:44.302968 16700 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:56:44.303864 16715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:56:44.304085 16700 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 21:56:44.304185 16700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-1/wal
uuid: "69a6ebbd4f834658854802f999deed03"
format_stamp: "Formatted at 2026-09-28 21:56:44 on dist-test-slave-k6lz"
server_key: "e7d421bfce3f3143d1eb912196f2d41d"
server_key_iv: "711d2ac6a7ace06e7156feee47c58646"
server_key_version: "encryptionkey@0"
I20260928 21:56:44.304262 16700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:56:44.326730 16700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:56:44.329084 16700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:56:44.329207 16700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:56:44.329442 16700 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:56:44.329761 16700 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:56:44.329793 16700 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:56:44.329815 16700 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:56:44.329838 16700 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:56:44.335901 16700 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.66:40155
I20260928 21:56:44.335935 16828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.66:40155 every 8 connection(s)
I20260928 21:56:44.336261 16700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
I20260928 21:56:44.339547 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 16700
I20260928 21:56:44.339648 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-1/wal/instance
I20260928 21:56:44.339846 2777 external_mini_cluster.cc:1428] Setting key cdfe0b95e4151b69fbc1bb0bbcd8fe37
Sep 28 21:56:44 dist-test-slave-k6lz krb5kdc[16459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632604, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.66@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:56:44.346285 16544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:44.336981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.66:56219 (local address 127.2.182.126:46141)
0928 21:56:44.337040 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 21:56:44.337042 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:56:44.337314 (+ 272us) server_negotiation.cc:407] Connection header received
0928 21:56:44.337764 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:56:44.337766 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:56:44.337778 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:56:44.337829 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:56:44.338768 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:44.339270 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:44.340423 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:44.340708 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:44.340767 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:44.343425 (+ 2658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:56:44.343427 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:56:44.343427 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:56:44.343434 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:56:44.344814 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:56:44.345309 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:56:44.345310 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:56:44.345311 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:56:44.345370 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:56:44.345647 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:56:44.345648 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:56:44.345648 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:56:44.345759 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:56:44.345810 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 21:56:44.346137 (+ 327us) server_negotiation.cc:299] Negotiation successful
0928 21:56:44.346215 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 21:56:44.346573 16831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:44.336961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:46141 (local address 127.2.182.66:56219)
0928 21:56:44.337114 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:44.337151 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 21:56:44.337661 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:56:44.337897 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:56:44.337901 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:56:44.338463 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:56:44.338664 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:56:44.338668 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:56:44.339382 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:56:44.339383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:56:44.340319 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:56:44.340323 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:56:44.340793 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:56:44.340795 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:56:44.340930 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:44.341438 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:56:44.341448 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:56:44.343278 (+ 1830us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:56:44.344900 (+ 1622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:56:44.344907 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:56:44.344910 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:56:44.345218 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:56:44.345442 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:56:44.345443 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:56:44.345443 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:56:44.345578 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:56:44.345820 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:56:44.345826 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:56:44.345989 (+ 163us) client_negotiation.cc:769] Sending connection context
0928 21:56:44.346110 (+ 121us) client_negotiation.cc:240] Negotiation successful
0928 21:56:44.346294 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":74,"threads_started":1}
I20260928 21:56:44.347091 16829 heartbeater.cc:347] Connected to a master server at 127.2.182.126:46141
I20260928 21:56:44.347208 16829 heartbeater.cc:464] Registering TS with master...
I20260928 21:56:44.347441 16829 heartbeater.cc:511] Master 127.2.182.126:46141 requested a full tablet report, sending...
I20260928 21:56:44.347869 16504 ts_manager.cc:194] Registered new tserver with Master: 69a6ebbd4f834658854802f999deed03 (127.2.182.66:40155)
WARNING: no policy specified for kudu/127.2.182.67@KRBTEST.COM; defaulting to no policy
I20260928 21:56:44.348273 16504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.66@KRBTEST.COM'} at 127.2.182.66:56219
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab.
Entry for principal kudu/127.2.182.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab.
I20260928 21:56:44.367049 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.67:0
--local_ip_for_outbound_sockets=127.2.182.67
--webserver_interface=127.2.182.67
--webserver_port=0
--tserver_master_addrs=127.2.182.126:46141
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab
--principal=kudu/127.2.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:33315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:56:44.455259 16834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:56:44.455508 16834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:56:44.455535 16834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:56:44.457114 16834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:56:44.457195 16834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:56:44.457970 16834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.67
I20260928 21:56:44.459980 16834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:33315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab
--principal=kudu/127.2.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.182.67
--webserver_port=0
--tserver_master_addrs=127.2.182.126:46141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.67
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:56:44.460261 16834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:56:44.460500 16834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:56:44.466064 16839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:56:44.466187 16840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:56:44.466089 16842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:56:44.466375 16834 server_base.cc:1034] running on GCE node
I20260928 21:56:44.466753 16834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:56:44.467092 16834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:56:44.468299 16834 hybrid_clock.cc:648] HybridClock initialized: now 1790632604468274 us; error 65 us; skew 500 ppm
Sep 28 21:56:44 dist-test-slave-k6lz krb5kdc[16459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632604, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:56:44.471979 16834 init.cc:377] Logged in from keytab as kudu/127.2.182.67@KRBTEST.COM (short username kudu)
I20260928 21:56:44.472579 16834 webserver.cc:460] Webserver started at http://127.2.182.67:36599/ using document root <none> and password file <none>
I20260928 21:56:44.472879 16834 fs_manager.cc:359] Metadata directory not provided
I20260928 21:56:44.472935 16834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:56:44.473060 16834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:56:44.474180 16834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-2/data/instance:
uuid: "35f5a5726eea46a988d9b74b3d6e4677"
format_stamp: "Formatted at 2026-09-28 21:56:44 on dist-test-slave-k6lz"
server_key: "91ad26c273c29d765986c82a4b03fcda"
server_key_iv: "8c48c0813c725fb0cdeb3e3a2679313e"
server_key_version: "encryptionkey@0"
I20260928 21:56:44.474521 16834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-2/wal/instance:
uuid: "35f5a5726eea46a988d9b74b3d6e4677"
format_stamp: "Formatted at 2026-09-28 21:56:44 on dist-test-slave-k6lz"
server_key: "91ad26c273c29d765986c82a4b03fcda"
server_key_iv: "8c48c0813c725fb0cdeb3e3a2679313e"
server_key_version: "encryptionkey@0"
I20260928 21:56:44.475915 16834 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:56:44.476783 16849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:56:44.476971 16834 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:56:44.477062 16834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-2/wal
uuid: "35f5a5726eea46a988d9b74b3d6e4677"
format_stamp: "Formatted at 2026-09-28 21:56:44 on dist-test-slave-k6lz"
server_key: "91ad26c273c29d765986c82a4b03fcda"
server_key_iv: "8c48c0813c725fb0cdeb3e3a2679313e"
server_key_version: "encryptionkey@0"
I20260928 21:56:44.477129 16834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:56:44.495635 16834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:56:44.498013 16834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:56:44.498162 16834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:56:44.498479 16834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:56:44.498797 16834 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:56:44.498829 16834 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:56:44.498859 16834 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:56:44.498884 16834 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:56:44.505280 16834 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.67:41977
I20260928 21:56:44.505427 16962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.67:41977 every 8 connection(s)
I20260928 21:56:44.505904 16834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
I20260928 21:56:44.512725 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 16834
I20260928 21:56:44.512840 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-2/wal/instance
I20260928 21:56:44.512998 2777 external_mini_cluster.cc:1428] Setting key bb870ce859e8b75c73ace2006129d6f0
Sep 28 21:56:44 dist-test-slave-k6lz krb5kdc[16459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632604, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.67@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:56:44.516692 16544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:44.506914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.67:35741 (local address 127.2.182.126:46141)
0928 21:56:44.506977 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 21:56:44.506979 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:56:44.507224 (+ 245us) server_negotiation.cc:407] Connection header received
0928 21:56:44.507824 (+ 600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:56:44.507825 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:56:44.507836 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:56:44.507906 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:56:44.508904 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:44.509489 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:44.510937 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:44.511149 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:44.511222 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:44.513779 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:56:44.513781 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:56:44.513781 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:56:44.513790 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:56:44.515315 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:56:44.515760 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:56:44.515762 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:56:44.515762 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:56:44.515823 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:56:44.516048 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:56:44.516049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:56:44.516049 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:56:44.516183 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:56:44.516239 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 21:56:44.516515 (+ 276us) server_negotiation.cc:299] Negotiation successful
0928 21:56:44.516597 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 21:56:44.516901 16965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:44.506869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:46141 (local address 127.2.182.67:35741)
0928 21:56:44.507048 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:44.507092 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 21:56:44.507720 (+ 628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:56:44.507962 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:56:44.507965 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:56:44.508581 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:56:44.508794 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:56:44.508799 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:56:44.509649 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:56:44.509650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:56:44.510820 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:56:44.510827 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:56:44.511270 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:56:44.511271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:56:44.511450 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:44.512096 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:56:44.512105 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:56:44.513678 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:56:44.515421 (+ 1743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:56:44.515430 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:56:44.515433 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:56:44.515689 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:56:44.515881 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:56:44.515881 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:56:44.515882 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:56:44.515988 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:56:44.516241 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:56:44.516247 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:56:44.516392 (+ 145us) client_negotiation.cc:769] Sending connection context
0928 21:56:44.516486 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 21:56:44.516671 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":87,"threads_started":1}
I20260928 21:56:44.517508 16963 heartbeater.cc:347] Connected to a master server at 127.2.182.126:46141
I20260928 21:56:44.517599 16963 heartbeater.cc:464] Registering TS with master...
I20260928 21:56:44.517759 16963 heartbeater.cc:511] Master 127.2.182.126:46141 requested a full tablet report, sending...
I20260928 21:56:44.518168 16504 ts_manager.cc:194] Registered new tserver with Master: 35f5a5726eea46a988d9b74b3d6e4677 (127.2.182.67:41977)
I20260928 21:56:44.518518 16504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.67@KRBTEST.COM'} at 127.2.182.67:35741
I20260928 21:56:44.524822 2777 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:56:44.533300 16544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:44.526153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60924 (local address 127.2.182.126:46141)
0928 21:56:44.526214 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 21:56:44.526254 (+ 40us) server_negotiation.cc:399] Waiting for connection header
0928 21:56:44.526267 (+ 13us) server_negotiation.cc:407] Connection header received
0928 21:56:44.526342 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:56:44.526342 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:56:44.526352 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:56:44.526406 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:56:44.526979 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:44.527603 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:44.528339 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:44.528529 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:44.528585 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:44.529912 (+ 1327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:56:44.529916 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:56:44.529916 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:56:44.529939 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:56:44.531723 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:56:44.532079 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:56:44.532081 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:56:44.532081 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:56:44.532144 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:56:44.532509 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:56:44.532510 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:56:44.532510 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:56:44.532680 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:56:44.532792 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 21:56:44.533087 (+ 295us) server_negotiation.cc:299] Negotiation successful
0928 21:56:44.533229 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 21:56:44.535772 16504 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60924:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:56:44.537784 16504 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:56:44.546703 16976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:44.541983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50858 (local address 127.2.182.66:40155)
0928 21:56:44.542305 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0928 21:56:44.542308 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:56:44.542404 (+ 96us) server_negotiation.cc:407] Connection header received
0928 21:56:44.542450 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:56:44.542452 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:56:44.542470 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:56:44.542640 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:56:44.542928 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:44.544297 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:44.545602 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:44.546377 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:44.546453 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:44.546508 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 21:56:44.546596 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 21:56:44.546630 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":91,"threads_started":1}
I20260928 21:56:44.546768 16975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:44.541845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.66:40155 (local address 127.0.0.1:50858)
0928 21:56:44.542292 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:44.542301 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 21:56:44.542345 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:56:44.542739 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:56:44.542742 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:56:44.542744 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:56:44.542842 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:56:44.542846 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:56:44.544426 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:56:44.544428 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:56:44.545495 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:56:44.545498 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:56:44.546461 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:56:44.546462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:56:44.546517 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:44.546521 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:56:44.546571 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 21:56:44.546608 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"mutex_wait_us":6,"thread_start_us":78,"threads_started":1}
I20260928 21:56:44.547152 16977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:44.541988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37306 (local address 127.2.182.67:41977)
0928 21:56:44.542184 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 21:56:44.542190 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:56:44.542203 (+ 13us) server_negotiation.cc:407] Connection header received
0928 21:56:44.542358 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:56:44.542361 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:56:44.542382 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:56:44.542777 (+ 395us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:56:44.543042 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:44.544020 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:44.545603 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:44.546377 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:44.546453 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:44.546507 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 21:56:44.547021 (+ 514us) server_negotiation.cc:299] Negotiation successful
0928 21:56:44.547073 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":89,"threads_started":1}
I20260928 21:56:44.546612 16978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:44.542074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.65:35749 (local address 127.0.0.1:35244)
0928 21:56:44.542639 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:44.542887 (+ 248us) client_negotiation.cc:174] Beginning negotiation
0928 21:56:44.542935 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:56:44.543164 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:56:44.543165 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:56:44.543168 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:56:44.543209 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:56:44.543210 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:56:44.544635 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:56:44.544636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:56:44.545461 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:56:44.545464 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:56:44.546266 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:56:44.546267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:56:44.546365 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:44.546381 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 21:56:44.546441 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 21:56:44.546503 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":67,"threads_started":1}
I20260928 21:56:44.547386 16974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:44.541846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.67:41977 (local address 127.0.0.1:37306)
0928 21:56:44.542050 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:44.542097 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 21:56:44.542179 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:56:44.542771 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:56:44.542773 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:56:44.542777 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:56:44.542932 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:56:44.542936 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:56:44.544164 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:56:44.544165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:56:44.545495 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:56:44.545498 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:56:44.546644 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:56:44.546645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:56:44.546698 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:44.546701 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:56:44.546745 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 21:56:44.546782 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":3,"spinlock_wait_cycles":1408,"thread_start_us":91,"threads_started":1}
I20260928 21:56:44.547619 16763 tablet_service.cc:1467] Processing CreateTablet for tablet dd29ef0fa93f48138eab4fc9c004f342 (DEFAULT_TABLE table=test-table [id=9bf17f7889084f0d9822bf6087dd5835]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:56:44.547888 16979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:44.542403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35244 (local address 127.2.182.65:35749)
0928 21:56:44.542593 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 21:56:44.542597 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:56:44.542988 (+ 391us) server_negotiation.cc:407] Connection header received
0928 21:56:44.543031 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:56:44.543033 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:56:44.543048 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:56:44.543141 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:56:44.543292 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:44.544529 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:44.545554 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:44.546186 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:44.546256 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:44.546324 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 21:56:44.546753 (+ 429us) server_negotiation.cc:299] Negotiation successful
0928 21:56:44.547376 (+ 623us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":86,"threads_started":1}
I20260928 21:56:44.547988 16763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd29ef0fa93f48138eab4fc9c004f342. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:56:44.548265 16897 tablet_service.cc:1467] Processing CreateTablet for tablet dd29ef0fa93f48138eab4fc9c004f342 (DEFAULT_TABLE table=test-table [id=9bf17f7889084f0d9822bf6087dd5835]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:56:44.548489 16629 tablet_service.cc:1467] Processing CreateTablet for tablet dd29ef0fa93f48138eab4fc9c004f342 (DEFAULT_TABLE table=test-table [id=9bf17f7889084f0d9822bf6087dd5835]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:56:44.548594 16897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd29ef0fa93f48138eab4fc9c004f342. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:56:44.548786 16629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd29ef0fa93f48138eab4fc9c004f342. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:56:44.551003 16980 tablet_bootstrap.cc:492] T dd29ef0fa93f48138eab4fc9c004f342 P 69a6ebbd4f834658854802f999deed03: Bootstrap starting.
I20260928 21:56:44.551743 16980 tablet_bootstrap.cc:654] T dd29ef0fa93f48138eab4fc9c004f342 P 69a6ebbd4f834658854802f999deed03: Neither blocks nor log segments found. Creating new log.
I20260928 21:56:44.552012 16981 tablet_bootstrap.cc:492] T dd29ef0fa93f48138eab4fc9c004f342 P 35f5a5726eea46a988d9b74b3d6e4677: Bootstrap starting.
I20260928 21:56:44.552112 16982 tablet_bootstrap.cc:492] T dd29ef0fa93f48138eab4fc9c004f342 P f91d4c3aa32644a28c325bfc0c55b068: Bootstrap starting.
I20260928 21:56:44.552340 16980 log.cc:824] T dd29ef0fa93f48138eab4fc9c004f342 P 69a6ebbd4f834658854802f999deed03: Log is configured to *not* fsync() on all Append() calls
I20260928 21:56:44.552675 16981 tablet_bootstrap.cc:654] T dd29ef0fa93f48138eab4fc9c004f342 P 35f5a5726eea46a988d9b74b3d6e4677: Neither blocks nor log segments found. Creating new log.
I20260928 21:56:44.552690 16982 tablet_bootstrap.cc:654] T dd29ef0fa93f48138eab4fc9c004f342 P f91d4c3aa32644a28c325bfc0c55b068: Neither blocks nor log segments found. Creating new log.
I20260928 21:56:44.553040 16980 tablet_bootstrap.cc:492] T dd29ef0fa93f48138eab4fc9c004f342 P 69a6ebbd4f834658854802f999deed03: No bootstrap required, opened a new log
I20260928 21:56:44.553102 16980 ts_tablet_manager.cc:1397] T dd29ef0fa93f48138eab4fc9c004f342 P 69a6ebbd4f834658854802f999deed03: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 21:56:44.553220 16982 log.cc:824] T dd29ef0fa93f48138eab4fc9c004f342 P f91d4c3aa32644a28c325bfc0c55b068: Log is configured to *not* fsync() on all Append() calls
I20260928 21:56:44.553244 16981 log.cc:824] T dd29ef0fa93f48138eab4fc9c004f342 P 35f5a5726eea46a988d9b74b3d6e4677: Log is configured to *not* fsync() on all Append() calls
I20260928 21:56:44.553912 16982 tablet_bootstrap.cc:492] T dd29ef0fa93f48138eab4fc9c004f342 P f91d4c3aa32644a28c325bfc0c55b068: No bootstrap required, opened a new log
I20260928 21:56:44.553921 16981 tablet_bootstrap.cc:492] T dd29ef0fa93f48138eab4fc9c004f342 P 35f5a5726eea46a988d9b74b3d6e4677: No bootstrap required, opened a new log
I20260928 21:56:44.553984 16982 ts_tablet_manager.cc:1397] T dd29ef0fa93f48138eab4fc9c004f342 P f91d4c3aa32644a28c325bfc0c55b068: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:56:44.553984 16981 ts_tablet_manager.cc:1397] T dd29ef0fa93f48138eab4fc9c004f342 P 35f5a5726eea46a988d9b74b3d6e4677: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 21:56:44.555042 16980 raft_consensus.cc:348] T dd29ef0fa93f48138eab4fc9c004f342 P 69a6ebbd4f834658854802f999deed03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f5a5726eea46a988d9b74b3d6e4677" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 41977 } } peers { permanent_uuid: "69a6ebbd4f834658854802f999deed03" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 40155 } } peers { permanent_uuid: "f91d4c3aa32644a28c325bfc0c55b068" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35749 } }
I20260928 21:56:44.555193 16980 raft_consensus.cc:374] T dd29ef0fa93f48138eab4fc9c004f342 P 69a6ebbd4f834658854802f999deed03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:56:44.555225 16980 raft_consensus.cc:729] T dd29ef0fa93f48138eab4fc9c004f342 P 69a6ebbd4f834658854802f999deed03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69a6ebbd4f834658854802f999deed03, State: Initialized, Role: FOLLOWER
I20260928 21:56:44.555397 16980 consensus_queue.cc:260] T dd29ef0fa93f48138eab4fc9c004f342 P 69a6ebbd4f834658854802f999deed03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f5a5726eea46a988d9b74b3d6e4677" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 41977 } } peers { permanent_uuid: "69a6ebbd4f834658854802f999deed03" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 40155 } } peers { permanent_uuid: "f91d4c3aa32644a28c325bfc0c55b068" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35749 } }
I20260928 21:56:44.555681 16980 ts_tablet_manager.cc:1428] T dd29ef0fa93f48138eab4fc9c004f342 P 69a6ebbd4f834658854802f999deed03: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 21:56:44.555819 16829 heartbeater.cc:503] Master 127.2.182.126:46141 was elected leader, sending a full tablet report...
I20260928 21:56:44.555970 16981 raft_consensus.cc:348] T dd29ef0fa93f48138eab4fc9c004f342 P 35f5a5726eea46a988d9b74b3d6e4677 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f5a5726eea46a988d9b74b3d6e4677" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 41977 } } peers { permanent_uuid: "69a6ebbd4f834658854802f999deed03" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 40155 } } peers { permanent_uuid: "f91d4c3aa32644a28c325bfc0c55b068" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35749 } }
I20260928 21:56:44.555970 16982 raft_consensus.cc:348] T dd29ef0fa93f48138eab4fc9c004f342 P f91d4c3aa32644a28c325bfc0c55b068 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f5a5726eea46a988d9b74b3d6e4677" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 41977 } } peers { permanent_uuid: "69a6ebbd4f834658854802f999deed03" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 40155 } } peers { permanent_uuid: "f91d4c3aa32644a28c325bfc0c55b068" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35749 } }
I20260928 21:56:44.556144 16981 raft_consensus.cc:374] T dd29ef0fa93f48138eab4fc9c004f342 P 35f5a5726eea46a988d9b74b3d6e4677 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:56:44.556144 16982 raft_consensus.cc:374] T dd29ef0fa93f48138eab4fc9c004f342 P f91d4c3aa32644a28c325bfc0c55b068 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:56:44.556185 16982 raft_consensus.cc:729] T dd29ef0fa93f48138eab4fc9c004f342 P f91d4c3aa32644a28c325bfc0c55b068 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f91d4c3aa32644a28c325bfc0c55b068, State: Initialized, Role: FOLLOWER
I20260928 21:56:44.556185 16981 raft_consensus.cc:729] T dd29ef0fa93f48138eab4fc9c004f342 P 35f5a5726eea46a988d9b74b3d6e4677 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35f5a5726eea46a988d9b74b3d6e4677, State: Initialized, Role: FOLLOWER
I20260928 21:56:44.556296 16981 consensus_queue.cc:260] T dd29ef0fa93f48138eab4fc9c004f342 P 35f5a5726eea46a988d9b74b3d6e4677 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f5a5726eea46a988d9b74b3d6e4677" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 41977 } } peers { permanent_uuid: "69a6ebbd4f834658854802f999deed03" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 40155 } } peers { permanent_uuid: "f91d4c3aa32644a28c325bfc0c55b068" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35749 } }
I20260928 21:56:44.556308 16982 consensus_queue.cc:260] T dd29ef0fa93f48138eab4fc9c004f342 P f91d4c3aa32644a28c325bfc0c55b068 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f5a5726eea46a988d9b74b3d6e4677" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 41977 } } peers { permanent_uuid: "69a6ebbd4f834658854802f999deed03" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 40155 } } peers { permanent_uuid: "f91d4c3aa32644a28c325bfc0c55b068" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35749 } }
I20260928 21:56:44.556558 16982 ts_tablet_manager.cc:1428] T dd29ef0fa93f48138eab4fc9c004f342 P f91d4c3aa32644a28c325bfc0c55b068: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 21:56:44.556558 16981 ts_tablet_manager.cc:1428] T dd29ef0fa93f48138eab4fc9c004f342 P 35f5a5726eea46a988d9b74b3d6e4677: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 21:56:44.556574 16963 heartbeater.cc:503] Master 127.2.182.126:46141 was elected leader, sending a full tablet report...
I20260928 21:56:44.556671 16695 heartbeater.cc:503] Master 127.2.182.126:46141 was elected leader, sending a full tablet report...
W20260928 21:56:44.586822 16830 tablet.cc:2369] T dd29ef0fa93f48138eab4fc9c004f342 P 69a6ebbd4f834658854802f999deed03: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:56:44.633947 16987 raft_consensus.cc:482] T dd29ef0fa93f48138eab4fc9c004f342 P 35f5a5726eea46a988d9b74b3d6e4677 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:56:44.634109 16987 raft_consensus.cc:504] T dd29ef0fa93f48138eab4fc9c004f342 P 35f5a5726eea46a988d9b74b3d6e4677 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f5a5726eea46a988d9b74b3d6e4677" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 41977 } } peers { permanent_uuid: "69a6ebbd4f834658854802f999deed03" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 40155 } } peers { permanent_uuid: "f91d4c3aa32644a28c325bfc0c55b068" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35749 } }
I20260928 21:56:44.634567 16987 leader_election.cc:288] T dd29ef0fa93f48138eab4fc9c004f342 P 35f5a5726eea46a988d9b74b3d6e4677 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 69a6ebbd4f834658854802f999deed03 (127.2.182.66:40155), f91d4c3aa32644a28c325bfc0c55b068 (127.2.182.65:35749)
I20260928 21:56:44.637485 16979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:44.635033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.67:47673 (local address 127.2.182.65:35749)
0928 21:56:44.635098 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 21:56:44.635100 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:56:44.635107 (+ 7us) server_negotiation.cc:407] Connection header received
0928 21:56:44.635130 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:56:44.635131 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:56:44.635139 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:56:44.635190 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:56:44.635347 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:44.635859 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:44.636754 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:44.637150 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:44.637220 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:44.637229 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 21:56:44.637376 (+ 147us) server_negotiation.cc:299] Negotiation successful
0928 21:56:44.637415 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 21:56:44.637494 16965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:44.634738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.66:40155 (local address 127.2.182.67:36139)
0928 21:56:44.634792 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:44.634804 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:56:44.634842 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:56:44.635010 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:56:44.635011 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:56:44.635013 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:56:44.635089 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:56:44.635090 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:56:44.635815 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:56:44.635816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:56:44.636601 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:56:44.636604 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:56:44.637170 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:56:44.637241 (+ 71us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:56:44.637326 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:44.637329 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:56:44.637382 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 21:56:44.637420 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":9}
I20260928 21:56:44.637506 16976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:44.634811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.67:36139 (local address 127.2.182.66:40155)
0928 21:56:44.634856 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 21:56:44.634858 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:56:44.634867 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:56:44.634894 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:56:44.634895 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:56:44.634904 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:56:44.634958 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:56:44.635170 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:44.635730 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:44.636693 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:44.637087 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:44.637139 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:44.637147 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 21:56:44.637405 (+ 258us) server_negotiation.cc:299] Negotiation successful
0928 21:56:44.637449 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 21:56:44.637501 16989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:44.634801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.65:35749 (local address 127.2.182.67:47673)
0928 21:56:44.635010 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:44.635020 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:56:44.635063 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:56:44.635202 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:56:44.635203 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:56:44.635204 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:56:44.635270 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:56:44.635272 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:56:44.635939 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:56:44.635939 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:56:44.636670 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:56:44.636672 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:56:44.637233 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:56:44.637234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:56:44.637305 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:44.637309 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:56:44.637352 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 21:56:44.637393 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":1,"thread_start_us":108,"threads_started":1}
I20260928 21:56:44.637830 16783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd29ef0fa93f48138eab4fc9c004f342" candidate_uuid: "35f5a5726eea46a988d9b74b3d6e4677" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69a6ebbd4f834658854802f999deed03" is_pre_election: true
I20260928 21:56:44.637872 16649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd29ef0fa93f48138eab4fc9c004f342" candidate_uuid: "35f5a5726eea46a988d9b74b3d6e4677" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f91d4c3aa32644a28c325bfc0c55b068" is_pre_election: true
I20260928 21:56:44.637997 16783 raft_consensus.cc:2454] T dd29ef0fa93f48138eab4fc9c004f342 P 69a6ebbd4f834658854802f999deed03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35f5a5726eea46a988d9b74b3d6e4677 in term 0.
I20260928 21:56:44.638001 16649 raft_consensus.cc:2454] T dd29ef0fa93f48138eab4fc9c004f342 P f91d4c3aa32644a28c325bfc0c55b068 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35f5a5726eea46a988d9b74b3d6e4677 in term 0.
I20260928 21:56:44.638202 16851 leader_election.cc:302] T dd29ef0fa93f48138eab4fc9c004f342 P 35f5a5726eea46a988d9b74b3d6e4677 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35f5a5726eea46a988d9b74b3d6e4677, 69a6ebbd4f834658854802f999deed03; no voters:
I20260928 21:56:44.638348 16987 raft_consensus.cc:2781] T dd29ef0fa93f48138eab4fc9c004f342 P 35f5a5726eea46a988d9b74b3d6e4677 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:56:44.638417 16987 raft_consensus.cc:482] T dd29ef0fa93f48138eab4fc9c004f342 P 35f5a5726eea46a988d9b74b3d6e4677 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:56:44.638440 16987 raft_consensus.cc:3037] T dd29ef0fa93f48138eab4fc9c004f342 P 35f5a5726eea46a988d9b74b3d6e4677 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:56:44.639138 16987 raft_consensus.cc:504] T dd29ef0fa93f48138eab4fc9c004f342 P 35f5a5726eea46a988d9b74b3d6e4677 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f5a5726eea46a988d9b74b3d6e4677" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 41977 } } peers { permanent_uuid: "69a6ebbd4f834658854802f999deed03" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 40155 } } peers { permanent_uuid: "f91d4c3aa32644a28c325bfc0c55b068" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35749 } }
I20260928 21:56:44.639283 16987 leader_election.cc:288] T dd29ef0fa93f48138eab4fc9c004f342 P 35f5a5726eea46a988d9b74b3d6e4677 [CANDIDATE]: Term 1 election: Requested vote from peers 69a6ebbd4f834658854802f999deed03 (127.2.182.66:40155), f91d4c3aa32644a28c325bfc0c55b068 (127.2.182.65:35749)
I20260928 21:56:44.639505 16783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd29ef0fa93f48138eab4fc9c004f342" candidate_uuid: "35f5a5726eea46a988d9b74b3d6e4677" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69a6ebbd4f834658854802f999deed03"
I20260928 21:56:44.639503 16649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd29ef0fa93f48138eab4fc9c004f342" candidate_uuid: "35f5a5726eea46a988d9b74b3d6e4677" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f91d4c3aa32644a28c325bfc0c55b068"
I20260928 21:56:44.639573 16649 raft_consensus.cc:3037] T dd29ef0fa93f48138eab4fc9c004f342 P f91d4c3aa32644a28c325bfc0c55b068 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:56:44.639576 16783 raft_consensus.cc:3037] T dd29ef0fa93f48138eab4fc9c004f342 P 69a6ebbd4f834658854802f999deed03 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:56:44.640228 16783 raft_consensus.cc:2454] T dd29ef0fa93f48138eab4fc9c004f342 P 69a6ebbd4f834658854802f999deed03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35f5a5726eea46a988d9b74b3d6e4677 in term 1.
I20260928 21:56:44.640374 16649 raft_consensus.cc:2454] T dd29ef0fa93f48138eab4fc9c004f342 P f91d4c3aa32644a28c325bfc0c55b068 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35f5a5726eea46a988d9b74b3d6e4677 in term 1.
I20260928 21:56:44.640396 16851 leader_election.cc:302] T dd29ef0fa93f48138eab4fc9c004f342 P 35f5a5726eea46a988d9b74b3d6e4677 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35f5a5726eea46a988d9b74b3d6e4677, 69a6ebbd4f834658854802f999deed03; no voters:
I20260928 21:56:44.640470 16987 raft_consensus.cc:2781] T dd29ef0fa93f48138eab4fc9c004f342 P 35f5a5726eea46a988d9b74b3d6e4677 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:56:44.640614 16987 raft_consensus.cc:686] T dd29ef0fa93f48138eab4fc9c004f342 P 35f5a5726eea46a988d9b74b3d6e4677 [term 1 LEADER]: Becoming Leader. State: Replica: 35f5a5726eea46a988d9b74b3d6e4677, State: Running, Role: LEADER
I20260928 21:56:44.640708 16987 consensus_queue.cc:237] T dd29ef0fa93f48138eab4fc9c004f342 P 35f5a5726eea46a988d9b74b3d6e4677 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f5a5726eea46a988d9b74b3d6e4677" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 41977 } } peers { permanent_uuid: "69a6ebbd4f834658854802f999deed03" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 40155 } } peers { permanent_uuid: "f91d4c3aa32644a28c325bfc0c55b068" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35749 } }
I20260928 21:56:44.641533 16502 catalog_manager.cc:5495] T dd29ef0fa93f48138eab4fc9c004f342 P 35f5a5726eea46a988d9b74b3d6e4677 reported cstate change: term changed from 0 to 1, leader changed from <none> to 35f5a5726eea46a988d9b74b3d6e4677 (127.2.182.67). New cstate: current_term: 1 leader_uuid: "35f5a5726eea46a988d9b74b3d6e4677" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f5a5726eea46a988d9b74b3d6e4677" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 41977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69a6ebbd4f834658854802f999deed03" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 40155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f91d4c3aa32644a28c325bfc0c55b068" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35749 } health_report { overall_health: UNKNOWN } } }
I20260928 21:56:44.662818 16977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:44.660493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37320 (local address 127.2.182.67:41977)
0928 21:56:44.660546 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 21:56:44.660548 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:56:44.660556 (+ 8us) server_negotiation.cc:407] Connection header received
0928 21:56:44.660651 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:56:44.660652 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:56:44.660673 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:56:44.660734 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:56:44.660859 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:44.661359 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:44.662104 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:44.662292 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:44.662349 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:44.662481 (+ 132us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:56:44.662591 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:56:44.662660 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 21:56:44.662697 (+ 37us) server_negotiation.cc:299] Negotiation successful
0928 21:56:44.662717 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 21:56:44.664746 16783 raft_consensus.cc:1261] T dd29ef0fa93f48138eab4fc9c004f342 P 69a6ebbd4f834658854802f999deed03 [term 1 FOLLOWER]: Refusing update from remote peer 35f5a5726eea46a988d9b74b3d6e4677: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:56:44.664758 16649 raft_consensus.cc:1261] T dd29ef0fa93f48138eab4fc9c004f342 P f91d4c3aa32644a28c325bfc0c55b068 [term 1 FOLLOWER]: Refusing update from remote peer 35f5a5726eea46a988d9b74b3d6e4677: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:56:44.665050 16990 consensus_queue.cc:1035] T dd29ef0fa93f48138eab4fc9c004f342 P 35f5a5726eea46a988d9b74b3d6e4677 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f91d4c3aa32644a28c325bfc0c55b068" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:56:44.665185 16987 consensus_queue.cc:1035] T dd29ef0fa93f48138eab4fc9c004f342 P 35f5a5726eea46a988d9b74b3d6e4677 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69a6ebbd4f834658854802f999deed03" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 40155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:56:44.667579 16993 mvcc.cc:205] Tried to move back new op lower bound from 7334431148703584256 to 7334431148608999424. Current Snapshot: MvccSnapshot[applied={T|T < 7334431148703584256}]
I20260928 21:56:44.667779 16995 mvcc.cc:205] Tried to move back new op lower bound from 7334431148703584256 to 7334431148608999424. Current Snapshot: MvccSnapshot[applied={T|T < 7334431148703584256}]
I20260928 21:56:44.670337 16504 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60924:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:56:44.670449 16504 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60924:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:56:44.671173 16504 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 73350ea0b7a94a888be60798c3d78cb2: Sending DeleteTablet for 3 replicas of tablet dd29ef0fa93f48138eab4fc9c004f342
I20260928 21:56:44.671531 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 16566
I20260928 21:56:44.671523 16763 tablet_service.cc:1514] Processing DeleteTablet for tablet dd29ef0fa93f48138eab4fc9c004f342 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:56:44 UTC) from {username='kudu', principal='kudu/127.2.182.126@KRBTEST.COM'} at 127.0.0.1:50858
I20260928 21:56:44.672004 16897 tablet_service.cc:1514] Processing DeleteTablet for tablet dd29ef0fa93f48138eab4fc9c004f342 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:56:44 UTC) from {username='kudu', principal='kudu/127.2.182.126@KRBTEST.COM'} at 127.0.0.1:37306
I20260928 21:56:44.673094 17004 tablet_replica.cc:330] T dd29ef0fa93f48138eab4fc9c004f342 P 35f5a5726eea46a988d9b74b3d6e4677: stopping tablet replica
I20260928 21:56:44.673202 17004 raft_consensus.cc:2229] T dd29ef0fa93f48138eab4fc9c004f342 P 35f5a5726eea46a988d9b74b3d6e4677 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:56:44.673327 17004 raft_consensus.cc:2258] T dd29ef0fa93f48138eab4fc9c004f342 P 35f5a5726eea46a988d9b74b3d6e4677 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:56:44.673781 17004 ts_tablet_manager.cc:1905] T dd29ef0fa93f48138eab4fc9c004f342 P 35f5a5726eea46a988d9b74b3d6e4677: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:56:44.675976 17004 ts_tablet_manager.cc:1918] T dd29ef0fa93f48138eab4fc9c004f342 P 35f5a5726eea46a988d9b74b3d6e4677: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:56:44.676056 17004 log.cc:1196] T dd29ef0fa93f48138eab4fc9c004f342 P 35f5a5726eea46a988d9b74b3d6e4677: Deleting WAL directory at /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-2/wal/wals/dd29ef0fa93f48138eab4fc9c004f342
I20260928 21:56:44.676333 17004 ts_tablet_manager.cc:1939] T dd29ef0fa93f48138eab4fc9c004f342 P 35f5a5726eea46a988d9b74b3d6e4677: Deleting consensus metadata
I20260928 21:56:44.676808 16488 catalog_manager.cc:4841] TS 35f5a5726eea46a988d9b74b3d6e4677 (127.2.182.67:41977): tablet dd29ef0fa93f48138eab4fc9c004f342 (table test-table [id=9bf17f7889084f0d9822bf6087dd5835]) successfully deleted
W20260928 21:56:44.677567 16488 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.182.65:35749 (error 108)
I20260928 21:56:44.678115 16974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:44.677974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.65:35749 (local address 127.0.0.1:35258)
0928 21:56:44.678034 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:44.678071 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.65:35749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 21:56:44.678154 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 16700
W20260928 21:56:44.678236 16488 catalog_manager.cc:4584] TS f91d4c3aa32644a28c325bfc0c55b068 (127.2.182.65:35749): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dd29ef0fa93f48138eab4fc9c004f342: Network error: Client connection negotiation failed: client connection to 127.2.182.65:35749: connect: Connection refused (error 111)
I20260928 21:56:44.683247 16974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:44.683107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.66:40155 (local address 127.0.0.1:50864)
0928 21:56:44.683173 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:44.683204 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.66:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 21:56:44.683310 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 16834
I20260928 21:56:44.687866 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 16472
I20260928 21:56:44.692746 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:46141
--webserver_interface=127.2.182.126
--webserver_port=42153
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab
--principal=kudu/127.2.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:33315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.126:46141
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:56:44.773561 17006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:56:44.773782 17006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:56:44.773803 17006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:56:44.774962 17006 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:56:44.775002 17006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:56:44.775014 17006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:56:44.775025 17006 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:56:44.775036 17006 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:56:44.776526 17006 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:33315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.2.182.126:46141
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab
--principal=kudu/127.2.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:46141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.126
--webserver_port=42153
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:56:44.776742 17006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:56:44.776968 17006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:56:44.785389 17015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:56:44.785390 17012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:56:44.785389 17013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:56:44.785796 17006 server_base.cc:1034] running on GCE node
I20260928 21:56:44.786065 17006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:56:44.786418 17006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:56:44.787597 17006 hybrid_clock.cc:648] HybridClock initialized: now 1790632604787569 us; error 44 us; skew 500 ppm
Sep 28 21:56:44 dist-test-slave-k6lz krb5kdc[16459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632604, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:56:44.796860 17006 init.cc:377] Logged in from keytab as kudu/127.2.182.126@KRBTEST.COM (short username kudu)
I20260928 21:56:44.797482 17006 webserver.cc:460] Webserver started at http://127.2.182.126:42153/ using document root <none> and password file <none>
I20260928 21:56:44.797717 17006 fs_manager.cc:359] Metadata directory not provided
I20260928 21:56:44.797778 17006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:56:44.799294 17006 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:56:44.800074 17022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:56:44.800356 17006 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:56:44.800454 17006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/master-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/master-0/wal
uuid: "73350ea0b7a94a888be60798c3d78cb2"
format_stamp: "Formatted at 2026-09-28 21:56:43 on dist-test-slave-k6lz"
server_key: "00124e733a592de95d6bf3109f613237"
server_key_iv: "5689f5249293130e727d1f0feb04321f"
server_key_version: "encryptionkey@0"
I20260928 21:56:44.800894 17006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:56:44.818085 17006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:56:44.820606 17006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:56:44.820765 17006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:56:44.824863 17006 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.126:46141
I20260928 21:56:44.824904 17074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.126:46141 every 8 connection(s)
I20260928 21:56:44.825274 17006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
I20260928 21:56:44.826947 17075 sys_catalog.cc:263] Verifying existing consensus state
I20260928 21:56:44.827613 17075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73350ea0b7a94a888be60798c3d78cb2: Bootstrap starting.
I20260928 21:56:44.827632 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 17006
I20260928 21:56:44.827873 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.65:35749
--local_ip_for_outbound_sockets=127.2.182.65
--tserver_master_addrs=127.2.182.126:46141
--webserver_port=37315
--webserver_interface=127.2.182.65
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab
--principal=kudu/127.2.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:33315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:56:44.835486 17075 log.cc:824] T 00000000000000000000000000000000 P 73350ea0b7a94a888be60798c3d78cb2: Log is configured to *not* fsync() on all Append() calls
I20260928 21:56:44.837637 17075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73350ea0b7a94a888be60798c3d78cb2: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 21:56:44.837931 17075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73350ea0b7a94a888be60798c3d78cb2: Bootstrap complete.
I20260928 21:56:44.840130 17075 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73350ea0b7a94a888be60798c3d78cb2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73350ea0b7a94a888be60798c3d78cb2" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 46141 } }
I20260928 21:56:44.840368 17075 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73350ea0b7a94a888be60798c3d78cb2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73350ea0b7a94a888be60798c3d78cb2, State: Initialized, Role: FOLLOWER
I20260928 21:56:44.840519 17075 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73350ea0b7a94a888be60798c3d78cb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73350ea0b7a94a888be60798c3d78cb2" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 46141 } }
I20260928 21:56:44.840591 17075 raft_consensus.cc:388] T 00000000000000000000000000000000 P 73350ea0b7a94a888be60798c3d78cb2 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:56:44.840646 17075 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73350ea0b7a94a888be60798c3d78cb2 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:56:44.840692 17075 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73350ea0b7a94a888be60798c3d78cb2 [term 1 FOLLOWER]: Advancing to term 2
I20260928 21:56:44.841703 17075 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73350ea0b7a94a888be60798c3d78cb2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73350ea0b7a94a888be60798c3d78cb2" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 46141 } }
I20260928 21:56:44.841797 17075 leader_election.cc:302] T 00000000000000000000000000000000 P 73350ea0b7a94a888be60798c3d78cb2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73350ea0b7a94a888be60798c3d78cb2; no voters:
I20260928 21:56:44.841957 17075 leader_election.cc:288] T 00000000000000000000000000000000 P 73350ea0b7a94a888be60798c3d78cb2 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 21:56:44.842051 17079 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73350ea0b7a94a888be60798c3d78cb2 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 21:56:44.842162 17079 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73350ea0b7a94a888be60798c3d78cb2 [term 2 LEADER]: Becoming Leader. State: Replica: 73350ea0b7a94a888be60798c3d78cb2, State: Running, Role: LEADER
I20260928 21:56:44.842252 17075 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73350ea0b7a94a888be60798c3d78cb2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:56:44.842252 17079 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73350ea0b7a94a888be60798c3d78cb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73350ea0b7a94a888be60798c3d78cb2" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 46141 } }
I20260928 21:56:44.842548 17080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73350ea0b7a94a888be60798c3d78cb2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "73350ea0b7a94a888be60798c3d78cb2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73350ea0b7a94a888be60798c3d78cb2" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 46141 } } }
I20260928 21:56:44.842550 17081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73350ea0b7a94a888be60798c3d78cb2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73350ea0b7a94a888be60798c3d78cb2. Latest consensus state: current_term: 2 leader_uuid: "73350ea0b7a94a888be60798c3d78cb2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73350ea0b7a94a888be60798c3d78cb2" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 46141 } } }
I20260928 21:56:44.842670 17080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73350ea0b7a94a888be60798c3d78cb2 [sys.catalog]: This master's current role is: LEADER
I20260928 21:56:44.842674 17081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73350ea0b7a94a888be60798c3d78cb2 [sys.catalog]: This master's current role is: LEADER
I20260928 21:56:44.842905 17084 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:56:44.843736 17084 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:56:44.843878 17084 catalog_manager.cc:1259] Loaded cluster ID: 71eab75250cc46ec98c3702a895ec83d
I20260928 21:56:44.843955 17084 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:56:44.844367 17084 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260928 21:56:44.848878 17084 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260928 21:56:44.851922 17084 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:56:44.852131 17084 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 73350ea0b7a94a888be60798c3d78cb2: Loaded TSK: 0
I20260928 21:56:44.852331 17084 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 21:56:44.912789 17077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:56:44.912999 17077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:56:44.913030 17077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:56:44.914253 17077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:56:44.914301 17077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:56:44.914327 17077 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.65
I20260928 21:56:44.915831 17077 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:33315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab
--principal=kudu/127.2.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.65:35749
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.182.65
--webserver_port=37315
--tserver_master_addrs=127.2.182.126:46141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17077
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.65
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:56:44.916195 17077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:56:44.916427 17077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:56:44.921553 17102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:56:44.921519 17103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:56:44.921671 17077 server_base.cc:1034] running on GCE node
W20260928 21:56:44.921711 17105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:56:44.921860 17077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:56:44.922156 17077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:56:44.923352 17077 hybrid_clock.cc:648] HybridClock initialized: now 1790632604923301 us; error 39 us; skew 500 ppm
Sep 28 21:56:44 dist-test-slave-k6lz krb5kdc[16459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632604, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:56:44.926026 17077 init.cc:377] Logged in from keytab as kudu/127.2.182.65@KRBTEST.COM (short username kudu)
I20260928 21:56:44.926604 17077 webserver.cc:460] Webserver started at http://127.2.182.65:37315/ using document root <none> and password file <none>
I20260928 21:56:44.926813 17077 fs_manager.cc:359] Metadata directory not provided
I20260928 21:56:44.926878 17077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:56:44.928267 17077 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:56:44.928853 17112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:56:44.929028 17077 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:56:44.929131 17077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-0/wal
uuid: "f91d4c3aa32644a28c325bfc0c55b068"
format_stamp: "Formatted at 2026-09-28 21:56:44 on dist-test-slave-k6lz"
server_key: "10b3104d0bdea2c40b77e84c40c1f9e1"
server_key_iv: "3595306e0995fcca1342728259a51ad5"
server_key_version: "encryptionkey@0"
I20260928 21:56:44.929579 17077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:56:44.952811 17077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:56:44.955451 17077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:56:44.955613 17077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:56:44.955900 17077 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:56:44.956450 17119 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 21:56:44.957890 17077 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 21:56:44.957952 17077 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 21:56:44.957985 17077 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 21:56:44.958825 17077 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 21:56:44.958869 17077 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 21:56:44.958959 17119 tablet_bootstrap.cc:492] T dd29ef0fa93f48138eab4fc9c004f342 P f91d4c3aa32644a28c325bfc0c55b068: Bootstrap starting.
I20260928 21:56:44.966284 17077 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.65:35749
I20260928 21:56:44.966812 17077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
I20260928 21:56:44.967502 17226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.65:35749 every 8 connection(s)
I20260928 21:56:44.969861 17119 log.cc:824] T dd29ef0fa93f48138eab4fc9c004f342 P f91d4c3aa32644a28c325bfc0c55b068: Log is configured to *not* fsync() on all Append() calls
I20260928 21:56:44.972496 17119 tablet_bootstrap.cc:492] T dd29ef0fa93f48138eab4fc9c004f342 P f91d4c3aa32644a28c325bfc0c55b068: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 21:56:44.972838 17119 tablet_bootstrap.cc:492] T dd29ef0fa93f48138eab4fc9c004f342 P f91d4c3aa32644a28c325bfc0c55b068: Bootstrap complete.
I20260928 21:56:44.972832 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 17077
I20260928 21:56:44.973080 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.66:40155
--local_ip_for_outbound_sockets=127.2.182.66
--tserver_master_addrs=127.2.182.126:46141
--webserver_port=45681
--webserver_interface=127.2.182.66
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab
--principal=kudu/127.2.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:33315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:56:44.974049 17119 ts_tablet_manager.cc:1397] T dd29ef0fa93f48138eab4fc9c004f342 P f91d4c3aa32644a28c325bfc0c55b068: Time spent bootstrapping tablet: real 0.015s user 0.012s sys 0.000s
Sep 28 21:56:44 dist-test-slave-k6lz krb5kdc[16459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632604, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.65@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:56:44.976379 17119 raft_consensus.cc:348] T dd29ef0fa93f48138eab4fc9c004f342 P f91d4c3aa32644a28c325bfc0c55b068 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f5a5726eea46a988d9b74b3d6e4677" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 41977 } } peers { permanent_uuid: "69a6ebbd4f834658854802f999deed03" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 40155 } } peers { permanent_uuid: "f91d4c3aa32644a28c325bfc0c55b068" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35749 } }
I20260928 21:56:44.976743 17119 raft_consensus.cc:729] T dd29ef0fa93f48138eab4fc9c004f342 P f91d4c3aa32644a28c325bfc0c55b068 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f91d4c3aa32644a28c325bfc0c55b068, State: Initialized, Role: FOLLOWER
I20260928 21:56:44.976908 17119 consensus_queue.cc:260] T dd29ef0fa93f48138eab4fc9c004f342 P f91d4c3aa32644a28c325bfc0c55b068 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f5a5726eea46a988d9b74b3d6e4677" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 41977 } } peers { permanent_uuid: "69a6ebbd4f834658854802f999deed03" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 40155 } } peers { permanent_uuid: "f91d4c3aa32644a28c325bfc0c55b068" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35749 } }
I20260928 21:56:44.977226 17119 ts_tablet_manager.cc:1428] T dd29ef0fa93f48138eab4fc9c004f342 P f91d4c3aa32644a28c325bfc0c55b068: Time spent starting tablet: real 0.003s user 0.005s sys 0.000s
I20260928 21:56:44.981176 17230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:44.968116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.65:40433 (local address 127.2.182.126:46141)
0928 21:56:44.968330 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 21:56:44.968341 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 21:56:44.968396 (+ 55us) server_negotiation.cc:407] Connection header received
0928 21:56:44.969070 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:56:44.969081 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:56:44.969140 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:56:44.969305 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:56:44.970138 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:44.971061 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:44.972124 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:44.972515 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:44.972615 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:44.977401 (+ 4786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:56:44.977405 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:56:44.977406 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:56:44.977419 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:56:44.979289 (+ 1870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:56:44.979814 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:56:44.979818 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:56:44.979818 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:56:44.979932 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:56:44.980200 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:56:44.980201 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:56:44.980202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:56:44.980499 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:56:44.980595 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 21:56:44.980923 (+ 328us) server_negotiation.cc:299] Negotiation successful
0928 21:56:44.981034 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":105,"threads_started":1}
I20260928 21:56:44.981241 17229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:44.967738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:46141 (local address 127.2.182.65:40433)
0928 21:56:44.967939 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:44.967983 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 21:56:44.968811 (+ 828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:56:44.969343 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:56:44.969348 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:56:44.969717 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:56:44.970008 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:56:44.970017 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:56:44.971188 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:56:44.971188 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:56:44.972034 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:56:44.972036 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:56:44.974285 (+ 2249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:56:44.974287 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:56:44.974413 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:44.974955 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:56:44.974964 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:56:44.977211 (+ 2247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:56:44.979400 (+ 2189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:56:44.979409 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:56:44.979411 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:56:44.979725 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:56:44.980001 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:56:44.980002 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:56:44.980002 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:56:44.980134 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:56:44.980573 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:56:44.980579 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:56:44.980747 (+ 168us) client_negotiation.cc:769] Sending connection context
0928 21:56:44.980870 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 21:56:44.981047 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":114,"threads_started":1}
I20260928 21:56:44.982025 17227 heartbeater.cc:347] Connected to a master server at 127.2.182.126:46141
I20260928 21:56:44.982113 17227 heartbeater.cc:464] Registering TS with master...
I20260928 21:56:44.982283 17227 heartbeater.cc:511] Master 127.2.182.126:46141 requested a full tablet report, sending...
I20260928 21:56:44.982936 17039 ts_manager.cc:194] Registered new tserver with Master: f91d4c3aa32644a28c325bfc0c55b068 (127.2.182.65:35749)
I20260928 21:56:44.984020 17039 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.65@KRBTEST.COM'} at 127.2.182.65:40433
I20260928 21:56:44.984795 17227 heartbeater.cc:503] Master 127.2.182.126:46141 was elected leader, sending a full tablet report...
Sep 28 21:56:44 dist-test-slave-k6lz krb5kdc[16459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632604, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.126@KRBTEST.COM for kudu/127.2.182.65@KRBTEST.COM
I20260928 21:56:44.992076 17235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:44.983724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35268 (local address 127.2.182.65:35749)
0928 21:56:44.984296 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0928 21:56:44.984300 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:56:44.984310 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:56:44.984341 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:56:44.984343 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:56:44.984356 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:56:44.984415 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:56:44.985401 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:44.986040 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:44.986767 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:44.987044 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:44.987107 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:44.989390 (+ 2283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:56:44.989399 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:56:44.989399 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:56:44.989406 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:56:44.990814 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:56:44.991230 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:56:44.991232 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:56:44.991233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:56:44.991289 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:56:44.991482 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:56:44.991483 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:56:44.991484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:56:44.991640 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:56:44.991717 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 21:56:44.991883 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 21:56:44.991975 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":65,"threads_started":1}
I20260928 21:56:44.992076 17234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:44.983513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.65:35749 (local address 127.0.0.1:35268)
0928 21:56:44.983708 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:44.983754 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 21:56:44.983820 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:56:44.984962 (+ 1142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:56:44.984969 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:56:44.985280 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:56:44.985332 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:56:44.985335 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:56:44.986112 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:56:44.986113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:56:44.986692 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:56:44.986694 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:56:44.987112 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:56:44.987113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:56:44.987187 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:44.987722 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:56:44.987733 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:56:44.989297 (+ 1564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:56:44.990910 (+ 1613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:56:44.990916 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:56:44.990918 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:56:44.991107 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:56:44.991369 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:56:44.991370 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:56:44.991370 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:56:44.991433 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:56:44.991710 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:56:44.991716 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:56:44.991771 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 21:56:44.991843 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 21:56:44.991975 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":107,"threads_started":1}
I20260928 21:56:44.992473 17161 tablet_service.cc:1514] Processing DeleteTablet for tablet dd29ef0fa93f48138eab4fc9c004f342 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:56:44 UTC) from {username='kudu', principal='kudu/127.2.182.126@KRBTEST.COM'} at 127.0.0.1:35268
I20260928 21:56:44.992972 17236 tablet_replica.cc:330] stopping tablet replica
I20260928 21:56:44.993083 17236 raft_consensus.cc:2229] T dd29ef0fa93f48138eab4fc9c004f342 P f91d4c3aa32644a28c325bfc0c55b068 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:56:44.993149 17236 raft_consensus.cc:2258] T dd29ef0fa93f48138eab4fc9c004f342 P f91d4c3aa32644a28c325bfc0c55b068 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:56:44.993458 17236 ts_tablet_manager.cc:1905] T dd29ef0fa93f48138eab4fc9c004f342 P f91d4c3aa32644a28c325bfc0c55b068: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:56:44.995913 17236 ts_tablet_manager.cc:1918] T dd29ef0fa93f48138eab4fc9c004f342 P f91d4c3aa32644a28c325bfc0c55b068: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:56:44.995985 17236 log.cc:1196] T dd29ef0fa93f48138eab4fc9c004f342 P f91d4c3aa32644a28c325bfc0c55b068: Deleting WAL directory at /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-0/wal/wals/dd29ef0fa93f48138eab4fc9c004f342
I20260928 21:56:44.996179 17236 ts_tablet_manager.cc:1939] T dd29ef0fa93f48138eab4fc9c004f342 P f91d4c3aa32644a28c325bfc0c55b068: Deleting consensus metadata
I20260928 21:56:44.996872 17023 catalog_manager.cc:4841] TS f91d4c3aa32644a28c325bfc0c55b068 (127.2.182.65:35749): tablet dd29ef0fa93f48138eab4fc9c004f342 (table test-table [id=9bf17f7889084f0d9822bf6087dd5835]) successfully deleted
W20260928 21:56:45.057433 17232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:56:45.057632 17232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:56:45.057654 17232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:56:45.058789 17232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:56:45.058833 17232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:56:45.058859 17232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.66
I20260928 21:56:45.060338 17232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:33315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab
--principal=kudu/127.2.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.66:40155
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.182.66
--webserver_port=45681
--tserver_master_addrs=127.2.182.126:46141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17232
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.66
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:56:45.060580 17232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:56:45.060791 17232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:56:45.065569 17241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:56:45.065604 17244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:56:45.065744 17232 server_base.cc:1034] running on GCE node
W20260928 21:56:45.065626 17242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:56:45.066041 17232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:56:45.066265 17232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:56:45.067382 17232 hybrid_clock.cc:648] HybridClock initialized: now 1790632605067371 us; error 28 us; skew 500 ppm
Sep 28 21:56:45 dist-test-slave-k6lz krb5kdc[16459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632605, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:56:45.069768 17232 init.cc:377] Logged in from keytab as kudu/127.2.182.66@KRBTEST.COM (short username kudu)
I20260928 21:56:45.070175 17232 webserver.cc:460] Webserver started at http://127.2.182.66:45681/ using document root <none> and password file <none>
I20260928 21:56:45.070386 17232 fs_manager.cc:359] Metadata directory not provided
I20260928 21:56:45.070447 17232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:56:45.071736 17232 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:56:45.072328 17251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:56:45.072492 17232 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 21:56:45.072567 17232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-1/wal
uuid: "69a6ebbd4f834658854802f999deed03"
format_stamp: "Formatted at 2026-09-28 21:56:44 on dist-test-slave-k6lz"
server_key: "e7d421bfce3f3143d1eb912196f2d41d"
server_key_iv: "711d2ac6a7ace06e7156feee47c58646"
server_key_version: "encryptionkey@0"
I20260928 21:56:45.072896 17232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:56:45.085682 17232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:56:45.087776 17232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:56:45.087913 17232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:56:45.088117 17232 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:56:45.088500 17258 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 21:56:45.090034 17232 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 21:56:45.090097 17232 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 21:56:45.090135 17232 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 21:56:45.090785 17232 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 21:56:45.090823 17232 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 21:56:45.090911 17258 tablet_bootstrap.cc:492] T dd29ef0fa93f48138eab4fc9c004f342 P 69a6ebbd4f834658854802f999deed03: Bootstrap starting.
I20260928 21:56:45.097478 17232 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.66:40155
I20260928 21:56:45.097581 17365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.66:40155 every 8 connection(s)
I20260928 21:56:45.097868 17232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
I20260928 21:56:45.099862 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 17232
I20260928 21:56:45.099869 17258 log.cc:824] T dd29ef0fa93f48138eab4fc9c004f342 P 69a6ebbd4f834658854802f999deed03: Log is configured to *not* fsync() on all Append() calls
I20260928 21:56:45.100143 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.67:41977
--local_ip_for_outbound_sockets=127.2.182.67
--tserver_master_addrs=127.2.182.126:46141
--webserver_port=36599
--webserver_interface=127.2.182.67
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab
--principal=kudu/127.2.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:33315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:56:45.102216 17258 tablet_bootstrap.cc:492] T dd29ef0fa93f48138eab4fc9c004f342 P 69a6ebbd4f834658854802f999deed03: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 21:56:45.102514 17258 tablet_bootstrap.cc:492] T dd29ef0fa93f48138eab4fc9c004f342 P 69a6ebbd4f834658854802f999deed03: Bootstrap complete.
I20260928 21:56:45.103477 17258 ts_tablet_manager.cc:1397] T dd29ef0fa93f48138eab4fc9c004f342 P 69a6ebbd4f834658854802f999deed03: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.007s
Sep 28 21:56:45 dist-test-slave-k6lz krb5kdc[16459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632605, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.66@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:56:45.104683 17258 raft_consensus.cc:348] T dd29ef0fa93f48138eab4fc9c004f342 P 69a6ebbd4f834658854802f999deed03 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f5a5726eea46a988d9b74b3d6e4677" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 41977 } } peers { permanent_uuid: "69a6ebbd4f834658854802f999deed03" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 40155 } } peers { permanent_uuid: "f91d4c3aa32644a28c325bfc0c55b068" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35749 } }
I20260928 21:56:45.105016 17258 raft_consensus.cc:729] T dd29ef0fa93f48138eab4fc9c004f342 P 69a6ebbd4f834658854802f999deed03 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69a6ebbd4f834658854802f999deed03, State: Initialized, Role: FOLLOWER
I20260928 21:56:45.105171 17258 consensus_queue.cc:260] T dd29ef0fa93f48138eab4fc9c004f342 P 69a6ebbd4f834658854802f999deed03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f5a5726eea46a988d9b74b3d6e4677" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 41977 } } peers { permanent_uuid: "69a6ebbd4f834658854802f999deed03" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 40155 } } peers { permanent_uuid: "f91d4c3aa32644a28c325bfc0c55b068" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35749 } }
I20260928 21:56:45.105440 17258 ts_tablet_manager.cc:1428] T dd29ef0fa93f48138eab4fc9c004f342 P 69a6ebbd4f834658854802f999deed03: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260928 21:56:45.108032 17230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:45.098600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.66:40739 (local address 127.2.182.126:46141)
0928 21:56:45.098665 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 21:56:45.098667 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:56:45.098873 (+ 206us) server_negotiation.cc:407] Connection header received
0928 21:56:45.099649 (+ 776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:56:45.099650 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:56:45.099669 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:56:45.099726 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:56:45.100513 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:45.101200 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:45.102646 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:45.102850 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:45.102907 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:45.105356 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:56:45.105358 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:56:45.105358 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:56:45.105366 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:56:45.106644 (+ 1278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:56:45.107021 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:56:45.107022 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:56:45.107023 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:56:45.107082 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:56:45.107307 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:56:45.107308 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:56:45.107308 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:56:45.107452 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:56:45.107507 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 21:56:45.107801 (+ 294us) server_negotiation.cc:299] Negotiation successful
0928 21:56:45.107897 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 21:56:45.108220 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:45.098537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:46141 (local address 127.2.182.66:40739)
0928 21:56:45.098714 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:45.098773 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 21:56:45.099517 (+ 744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:56:45.099796 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:56:45.099800 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:56:45.100141 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:56:45.100408 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:56:45.100415 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:56:45.101322 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:56:45.101323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:56:45.102536 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:56:45.102539 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:56:45.102922 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:56:45.102922 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:56:45.103079 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:45.103680 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:56:45.103688 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:56:45.105237 (+ 1549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:56:45.106720 (+ 1483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:56:45.106727 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:56:45.106729 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:56:45.106954 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:56:45.107133 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:56:45.107134 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:56:45.107134 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:56:45.107250 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:56:45.107506 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:56:45.107512 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:56:45.107669 (+ 157us) client_negotiation.cc:769] Sending connection context
0928 21:56:45.107770 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 21:56:45.107932 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":74,"threads_started":1}
I20260928 21:56:45.108633 17366 heartbeater.cc:347] Connected to a master server at 127.2.182.126:46141
I20260928 21:56:45.108758 17366 heartbeater.cc:464] Registering TS with master...
I20260928 21:56:45.108929 17366 heartbeater.cc:511] Master 127.2.182.126:46141 requested a full tablet report, sending...
I20260928 21:56:45.109448 17039 ts_manager.cc:194] Registered new tserver with Master: 69a6ebbd4f834658854802f999deed03 (127.2.182.66:40155)
I20260928 21:56:45.109923 17039 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.66@KRBTEST.COM'} at 127.2.182.66:40739
I20260928 21:56:45.110616 17366 heartbeater.cc:503] Master 127.2.182.126:46141 was elected leader, sending a full tablet report...
Sep 28 21:56:45 dist-test-slave-k6lz krb5kdc[16459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632604, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.126@KRBTEST.COM for kudu/127.2.182.66@KRBTEST.COM
I20260928 21:56:45.117226 17234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:45.109767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.66:40155 (local address 127.0.0.1:50872)
0928 21:56:45.109820 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:45.109830 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:56:45.109867 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:56:45.110153 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:56:45.110153 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:56:45.110314 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:56:45.110366 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:56:45.110369 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:56:45.111360 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:56:45.111361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:56:45.112201 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:56:45.112203 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:56:45.112686 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:56:45.112687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:56:45.112751 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:45.113117 (+ 366us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:56:45.113122 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:56:45.114425 (+ 1303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:56:45.116332 (+ 1907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:56:45.116333 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:56:45.116334 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:56:45.116488 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:56:45.116673 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:56:45.116674 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:56:45.116674 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:56:45.116726 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:56:45.116993 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:56:45.116994 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:56:45.117028 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 21:56:45.117071 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 21:56:45.117148 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 21:56:45.117291 17372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:45.109830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50872 (local address 127.2.182.66:40155)
0928 21:56:45.109983 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 21:56:45.109987 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:56:45.109997 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:56:45.110030 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:56:45.110032 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:56:45.110044 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:56:45.110143 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:56:45.110446 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:45.111260 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:45.112319 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:45.112619 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:45.112672 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:45.114502 (+ 1830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:56:45.114511 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:56:45.114511 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:56:45.114518 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:56:45.116252 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:56:45.116552 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:56:45.116555 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:56:45.116556 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:56:45.116616 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:56:45.116773 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:56:45.116774 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:56:45.116774 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:56:45.116942 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:56:45.117018 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 21:56:45.117105 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 21:56:45.117197 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":78,"threads_started":1}
I20260928 21:56:45.117619 17300 tablet_service.cc:1514] Processing DeleteTablet for tablet dd29ef0fa93f48138eab4fc9c004f342 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:56:44 UTC) from {username='kudu', principal='kudu/127.2.182.126@KRBTEST.COM'} at 127.0.0.1:50872
I20260928 21:56:45.118170 17373 tablet_replica.cc:330] stopping tablet replica
I20260928 21:56:45.118268 17373 raft_consensus.cc:2229] T dd29ef0fa93f48138eab4fc9c004f342 P 69a6ebbd4f834658854802f999deed03 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:56:45.118340 17373 raft_consensus.cc:2258] T dd29ef0fa93f48138eab4fc9c004f342 P 69a6ebbd4f834658854802f999deed03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:56:45.118608 17373 ts_tablet_manager.cc:1905] T dd29ef0fa93f48138eab4fc9c004f342 P 69a6ebbd4f834658854802f999deed03: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:56:45.120559 17373 ts_tablet_manager.cc:1918] T dd29ef0fa93f48138eab4fc9c004f342 P 69a6ebbd4f834658854802f999deed03: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:56:45.120618 17373 log.cc:1196] T dd29ef0fa93f48138eab4fc9c004f342 P 69a6ebbd4f834658854802f999deed03: Deleting WAL directory at /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-1/wal/wals/dd29ef0fa93f48138eab4fc9c004f342
I20260928 21:56:45.120817 17373 ts_tablet_manager.cc:1939] T dd29ef0fa93f48138eab4fc9c004f342 P 69a6ebbd4f834658854802f999deed03: Deleting consensus metadata
I20260928 21:56:45.121268 17024 catalog_manager.cc:4841] TS 69a6ebbd4f834658854802f999deed03 (127.2.182.66:40155): tablet dd29ef0fa93f48138eab4fc9c004f342 (table test-table [id=9bf17f7889084f0d9822bf6087dd5835]) successfully deleted
W20260928 21:56:45.184585 17370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:56:45.184788 17370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:56:45.184819 17370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:56:45.185950 17370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:56:45.185988 17370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:56:45.186023 17370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.67
I20260928 21:56:45.187438 17370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:33315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab
--principal=kudu/127.2.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.67:41977
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.182.67
--webserver_port=36599
--tserver_master_addrs=127.2.182.126:46141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.67
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:56:45.187664 17370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:56:45.187856 17370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:56:45.192626 17378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:56:45.192673 17370 server_base.cc:1034] running on GCE node
W20260928 21:56:45.192641 17381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:56:45.192641 17379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:56:45.193020 17370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:56:45.193262 17370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:56:45.194440 17370 hybrid_clock.cc:648] HybridClock initialized: now 1790632605194423 us; error 26 us; skew 500 ppm
Sep 28 21:56:45 dist-test-slave-k6lz krb5kdc[16459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632605, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:56:45.197049 17370 init.cc:377] Logged in from keytab as kudu/127.2.182.67@KRBTEST.COM (short username kudu)
I20260928 21:56:45.197531 17370 webserver.cc:460] Webserver started at http://127.2.182.67:36599/ using document root <none> and password file <none>
I20260928 21:56:45.197767 17370 fs_manager.cc:359] Metadata directory not provided
I20260928 21:56:45.197831 17370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:56:45.199229 17370 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:56:45.199916 17388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:56:45.200079 17370 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 21:56:45.200162 17370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-2/wal
uuid: "35f5a5726eea46a988d9b74b3d6e4677"
format_stamp: "Formatted at 2026-09-28 21:56:44 on dist-test-slave-k6lz"
server_key: "91ad26c273c29d765986c82a4b03fcda"
server_key_iv: "8c48c0813c725fb0cdeb3e3a2679313e"
server_key_version: "encryptionkey@0"
I20260928 21:56:45.200577 17370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:56:45.224632 17370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:56:45.226940 17370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:56:45.227075 17370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:56:45.227454 17370 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:56:45.227756 17370 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:56:45.227788 17370 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:56:45.227818 17370 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:56:45.227854 17370 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:56:45.233620 17370 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.67:41977
I20260928 21:56:45.233670 17501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.67:41977 every 8 connection(s)
I20260928 21:56:45.234009 17370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
I20260928 21:56:45.236670 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 17370
Sep 28 21:56:45 dist-test-slave-k6lz krb5kdc[16459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632605, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.67@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:56:45.243858 17506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:45.237008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60926 (local address 127.2.182.126:46141)
0928 21:56:45.237349 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0928 21:56:45.237350 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:56:45.237356 (+ 6us) server_negotiation.cc:407] Connection header received
0928 21:56:45.237375 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:56:45.237375 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:56:45.237385 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:56:45.237441 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:56:45.237938 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:45.238588 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:45.239320 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:45.239533 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:45.239584 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:45.240868 (+ 1284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:56:45.240869 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:56:45.240870 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:56:45.240877 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:56:45.242452 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:56:45.242833 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:56:45.242834 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:56:45.242835 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:56:45.242894 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:56:45.243233 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:56:45.243235 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:56:45.243235 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:56:45.243450 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:56:45.243583 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 21:56:45.243676 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 21:56:45.243760 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":46,"threads_started":1}
I20260928 21:56:45.244370 17230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:45.234746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.67:44881 (local address 127.2.182.126:46141)
0928 21:56:45.234798 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 21:56:45.234800 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:56:45.234970 (+ 170us) server_negotiation.cc:407] Connection header received
0928 21:56:45.235578 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:56:45.235579 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:56:45.235592 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:56:45.235639 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:56:45.236591 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:45.237242 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:45.238464 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:45.238705 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:45.238775 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:45.241215 (+ 2440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:56:45.241216 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:56:45.241217 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:56:45.241224 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:56:45.242599 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:56:45.243171 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:56:45.243173 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:56:45.243173 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:56:45.243237 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:56:45.243516 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:56:45.243517 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:56:45.243517 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:56:45.243676 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:56:45.243741 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 21:56:45.244204 (+ 463us) server_negotiation.cc:299] Negotiation successful
0928 21:56:45.244289 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 21:56:45.244547 17504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:45.234722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:46141 (local address 127.2.182.67:44881)
0928 21:56:45.234858 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:45.234897 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 21:56:45.235481 (+ 584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:56:45.235717 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:56:45.235721 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:56:45.236289 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:56:45.236503 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:56:45.236507 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:56:45.237328 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:56:45.237329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:56:45.238372 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:56:45.238375 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:56:45.238808 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:56:45.238809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:56:45.238920 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:45.239456 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:56:45.239464 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:56:45.241126 (+ 1662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:56:45.242741 (+ 1615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:56:45.242750 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:56:45.242753 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:56:45.243113 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:56:45.243296 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:56:45.243297 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:56:45.243297 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:56:45.243446 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:56:45.243739 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:56:45.243745 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:56:45.243925 (+ 180us) client_negotiation.cc:769] Sending connection context
0928 21:56:45.244164 (+ 239us) client_negotiation.cc:240] Negotiation successful
0928 21:56:45.244314 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":69,"threads_started":1}
I20260928 21:56:45.244951 17502 heartbeater.cc:347] Connected to a master server at 127.2.182.126:46141
I20260928 21:56:45.245048 17502 heartbeater.cc:464] Registering TS with master...
I20260928 21:56:45.245240 17502 heartbeater.cc:511] Master 127.2.182.126:46141 requested a full tablet report, sending...
I20260928 21:56:45.245648 17039 ts_manager.cc:194] Registered new tserver with Master: 35f5a5726eea46a988d9b74b3d6e4677 (127.2.182.67:41977)
I20260928 21:56:45.245998 17039 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.67@KRBTEST.COM'} at 127.2.182.67:44881
I20260928 21:56:45.255897 2777 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:56:45.258663 17230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:45.256253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60936 (local address 127.2.182.126:46141)
0928 21:56:45.256293 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 21:56:45.256295 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:56:45.256381 (+ 86us) server_negotiation.cc:407] Connection header received
0928 21:56:45.256446 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:56:45.256447 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:56:45.256484 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:56:45.256549 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:56:45.256638 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:45.257191 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:45.257993 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:45.258191 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:45.258251 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:45.258392 (+ 141us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:56:45.258489 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:56:45.258548 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 21:56:45.258584 (+ 36us) server_negotiation.cc:299] Negotiation successful
0928 21:56:45.258600 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 21:56:45.258973 17039 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:60936:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:56:45.260658 17039 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:56:45.262856 17300 tablet_service.cc:1467] Processing CreateTablet for tablet 4b929080d9174670a57c130fef9db996 (DEFAULT_TABLE table=test-table [id=44bffbd9e4dd4b4093cad813dfdf9bf6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:56:45.262946 17161 tablet_service.cc:1467] Processing CreateTablet for tablet 4b929080d9174670a57c130fef9db996 (DEFAULT_TABLE table=test-table [id=44bffbd9e4dd4b4093cad813dfdf9bf6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:56:45.263054 17300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b929080d9174670a57c130fef9db996. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:56:45.263111 17161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b929080d9174670a57c130fef9db996. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:56:45.264592 17258 tablet_bootstrap.cc:492] T 4b929080d9174670a57c130fef9db996 P 69a6ebbd4f834658854802f999deed03: Bootstrap starting.
I20260928 21:56:45.264937 17258 tablet_bootstrap.cc:654] T 4b929080d9174670a57c130fef9db996 P 69a6ebbd4f834658854802f999deed03: Neither blocks nor log segments found. Creating new log.
I20260928 21:56:45.265190 17119 tablet_bootstrap.cc:492] T 4b929080d9174670a57c130fef9db996 P f91d4c3aa32644a28c325bfc0c55b068: Bootstrap starting.
I20260928 21:56:45.265483 17119 tablet_bootstrap.cc:654] T 4b929080d9174670a57c130fef9db996 P f91d4c3aa32644a28c325bfc0c55b068: Neither blocks nor log segments found. Creating new log.
I20260928 21:56:45.265497 17258 tablet_bootstrap.cc:492] T 4b929080d9174670a57c130fef9db996 P 69a6ebbd4f834658854802f999deed03: No bootstrap required, opened a new log
I20260928 21:56:45.265535 17258 ts_tablet_manager.cc:1397] T 4b929080d9174670a57c130fef9db996 P 69a6ebbd4f834658854802f999deed03: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260928 21:56:45.265872 17258 raft_consensus.cc:348] T 4b929080d9174670a57c130fef9db996 P 69a6ebbd4f834658854802f999deed03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f5a5726eea46a988d9b74b3d6e4677" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 41977 } } peers { permanent_uuid: "f91d4c3aa32644a28c325bfc0c55b068" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35749 } } peers { permanent_uuid: "69a6ebbd4f834658854802f999deed03" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 40155 } }
I20260928 21:56:45.265956 17258 raft_consensus.cc:374] T 4b929080d9174670a57c130fef9db996 P 69a6ebbd4f834658854802f999deed03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:56:45.266010 17258 raft_consensus.cc:729] T 4b929080d9174670a57c130fef9db996 P 69a6ebbd4f834658854802f999deed03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69a6ebbd4f834658854802f999deed03, State: Initialized, Role: FOLLOWER
I20260928 21:56:45.266052 17258 consensus_queue.cc:260] T 4b929080d9174670a57c130fef9db996 P 69a6ebbd4f834658854802f999deed03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f5a5726eea46a988d9b74b3d6e4677" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 41977 } } peers { permanent_uuid: "f91d4c3aa32644a28c325bfc0c55b068" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35749 } } peers { permanent_uuid: "69a6ebbd4f834658854802f999deed03" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 40155 } }
I20260928 21:56:45.266146 17258 ts_tablet_manager.cc:1428] T 4b929080d9174670a57c130fef9db996 P 69a6ebbd4f834658854802f999deed03: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 21:56:45.266937 17234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:45.262684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.67:41977 (local address 127.0.0.1:37326)
0928 21:56:45.262786 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:45.262796 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:56:45.262832 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:56:45.263320 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:56:45.263321 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:56:45.263323 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:56:45.263405 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:56:45.263407 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:56:45.264436 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:56:45.264438 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:56:45.265624 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:56:45.265628 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:56:45.266683 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:56:45.266684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:56:45.266750 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:45.266770 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 21:56:45.266821 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 21:56:45.266865 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260928 21:56:45.266986 17508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:45.262915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37326 (local address 127.2.182.67:41977)
0928 21:56:45.263149 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 21:56:45.263155 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:56:45.263165 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:56:45.263205 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:56:45.263207 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:56:45.263222 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:56:45.263301 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:56:45.263467 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:45.264339 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:45.265853 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:45.266613 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:45.266679 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:45.266747 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 21:56:45.266856 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 21:56:45.266905 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":108,"threads_started":1}
I20260928 21:56:45.267130 17119 tablet_bootstrap.cc:492] T 4b929080d9174670a57c130fef9db996 P f91d4c3aa32644a28c325bfc0c55b068: No bootstrap required, opened a new log
I20260928 21:56:45.267161 17119 ts_tablet_manager.cc:1397] T 4b929080d9174670a57c130fef9db996 P f91d4c3aa32644a28c325bfc0c55b068: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:56:45.267544 17119 raft_consensus.cc:348] T 4b929080d9174670a57c130fef9db996 P f91d4c3aa32644a28c325bfc0c55b068 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f5a5726eea46a988d9b74b3d6e4677" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 41977 } } peers { permanent_uuid: "f91d4c3aa32644a28c325bfc0c55b068" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35749 } } peers { permanent_uuid: "69a6ebbd4f834658854802f999deed03" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 40155 } }
I20260928 21:56:45.267617 17119 raft_consensus.cc:374] T 4b929080d9174670a57c130fef9db996 P f91d4c3aa32644a28c325bfc0c55b068 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:56:45.267647 17119 raft_consensus.cc:729] T 4b929080d9174670a57c130fef9db996 P f91d4c3aa32644a28c325bfc0c55b068 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f91d4c3aa32644a28c325bfc0c55b068, State: Initialized, Role: FOLLOWER
I20260928 21:56:45.267680 17119 consensus_queue.cc:260] T 4b929080d9174670a57c130fef9db996 P f91d4c3aa32644a28c325bfc0c55b068 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f5a5726eea46a988d9b74b3d6e4677" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 41977 } } peers { permanent_uuid: "f91d4c3aa32644a28c325bfc0c55b068" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35749 } } peers { permanent_uuid: "69a6ebbd4f834658854802f999deed03" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 40155 } }
I20260928 21:56:45.267782 17119 ts_tablet_manager.cc:1428] T 4b929080d9174670a57c130fef9db996 P f91d4c3aa32644a28c325bfc0c55b068: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 21:56:45.267860 17436 tablet_service.cc:1467] Processing CreateTablet for tablet 4b929080d9174670a57c130fef9db996 (DEFAULT_TABLE table=test-table [id=44bffbd9e4dd4b4093cad813dfdf9bf6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:56:45.268244 17436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b929080d9174670a57c130fef9db996. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:56:45.270882 17511 tablet_bootstrap.cc:492] T 4b929080d9174670a57c130fef9db996 P 35f5a5726eea46a988d9b74b3d6e4677: Bootstrap starting.
I20260928 21:56:45.271497 17511 tablet_bootstrap.cc:654] T 4b929080d9174670a57c130fef9db996 P 35f5a5726eea46a988d9b74b3d6e4677: Neither blocks nor log segments found. Creating new log.
I20260928 21:56:45.271926 17511 log.cc:824] T 4b929080d9174670a57c130fef9db996 P 35f5a5726eea46a988d9b74b3d6e4677: Log is configured to *not* fsync() on all Append() calls
I20260928 21:56:45.272522 17511 tablet_bootstrap.cc:492] T 4b929080d9174670a57c130fef9db996 P 35f5a5726eea46a988d9b74b3d6e4677: No bootstrap required, opened a new log
I20260928 21:56:45.272573 17511 ts_tablet_manager.cc:1397] T 4b929080d9174670a57c130fef9db996 P 35f5a5726eea46a988d9b74b3d6e4677: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:56:45.273963 17511 raft_consensus.cc:348] T 4b929080d9174670a57c130fef9db996 P 35f5a5726eea46a988d9b74b3d6e4677 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f5a5726eea46a988d9b74b3d6e4677" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 41977 } } peers { permanent_uuid: "f91d4c3aa32644a28c325bfc0c55b068" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35749 } } peers { permanent_uuid: "69a6ebbd4f834658854802f999deed03" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 40155 } }
I20260928 21:56:45.274134 17511 raft_consensus.cc:374] T 4b929080d9174670a57c130fef9db996 P 35f5a5726eea46a988d9b74b3d6e4677 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:56:45.274158 17511 raft_consensus.cc:729] T 4b929080d9174670a57c130fef9db996 P 35f5a5726eea46a988d9b74b3d6e4677 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35f5a5726eea46a988d9b74b3d6e4677, State: Initialized, Role: FOLLOWER
I20260928 21:56:45.274255 17511 consensus_queue.cc:260] T 4b929080d9174670a57c130fef9db996 P 35f5a5726eea46a988d9b74b3d6e4677 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f5a5726eea46a988d9b74b3d6e4677" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 41977 } } peers { permanent_uuid: "f91d4c3aa32644a28c325bfc0c55b068" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35749 } } peers { permanent_uuid: "69a6ebbd4f834658854802f999deed03" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 40155 } }
I20260928 21:56:45.274494 17511 ts_tablet_manager.cc:1428] T 4b929080d9174670a57c130fef9db996 P 35f5a5726eea46a988d9b74b3d6e4677: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 21:56:45.274520 17502 heartbeater.cc:503] Master 127.2.182.126:46141 was elected leader, sending a full tablet report...
W20260928 21:56:45.348412 17367 tablet.cc:2369] T 4b929080d9174670a57c130fef9db996 P 69a6ebbd4f834658854802f999deed03: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 21:56:45.468118 17228 tablet.cc:2369] T 4b929080d9174670a57c130fef9db996 P f91d4c3aa32644a28c325bfc0c55b068: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 21:56:45.484598 17503 tablet.cc:2369] T 4b929080d9174670a57c130fef9db996 P 35f5a5726eea46a988d9b74b3d6e4677: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:56:45.503041 17513 raft_consensus.cc:482] T 4b929080d9174670a57c130fef9db996 P 35f5a5726eea46a988d9b74b3d6e4677 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:56:45.503234 17513 raft_consensus.cc:504] T 4b929080d9174670a57c130fef9db996 P 35f5a5726eea46a988d9b74b3d6e4677 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f5a5726eea46a988d9b74b3d6e4677" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 41977 } } peers { permanent_uuid: "f91d4c3aa32644a28c325bfc0c55b068" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35749 } } peers { permanent_uuid: "69a6ebbd4f834658854802f999deed03" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 40155 } }
I20260928 21:56:45.503677 17513 leader_election.cc:288] T 4b929080d9174670a57c130fef9db996 P 35f5a5726eea46a988d9b74b3d6e4677 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f91d4c3aa32644a28c325bfc0c55b068 (127.2.182.65:35749), 69a6ebbd4f834658854802f999deed03 (127.2.182.66:40155)
I20260928 21:56:45.506536 17504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:45.503858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.66:40155 (local address 127.2.182.67:58509)
0928 21:56:45.503903 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:45.503915 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:56:45.503956 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:56:45.504141 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:56:45.504142 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:56:45.504144 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:56:45.504204 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:56:45.504205 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:56:45.504989 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:56:45.504990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:56:45.505732 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:56:45.505734 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:56:45.506285 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:56:45.506286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:56:45.506375 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:45.506379 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:56:45.506422 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 21:56:45.506462 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25,"mutex_wait_us":1}
I20260928 21:56:45.506568 17372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:45.503911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.67:58509 (local address 127.2.182.66:40155)
0928 21:56:45.503954 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 21:56:45.503977 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0928 21:56:45.503989 (+ 12us) server_negotiation.cc:407] Connection header received
0928 21:56:45.504019 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:56:45.504020 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:56:45.504044 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:56:45.504112 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:56:45.504274 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:45.504917 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:45.505817 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:45.506213 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:45.506285 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:45.506359 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 21:56:45.506441 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 21:56:45.506511 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 21:56:45.506744 17515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:45.503862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.65:35749 (local address 127.2.182.67:40389)
0928 21:56:45.504137 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:45.504146 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 21:56:45.504298 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:56:45.504469 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:56:45.504469 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:56:45.504471 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:56:45.504540 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:56:45.504542 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:56:45.505176 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:56:45.505176 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:56:45.505898 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:56:45.505900 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:56:45.506494 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:56:45.506495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:56:45.506571 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:45.506576 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 21:56:45.506624 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 21:56:45.506666 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"mutex_wait_us":139,"thread_start_us":86,"threads_started":1}
I20260928 21:56:45.506888 17514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:45.503979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.67:40389 (local address 127.2.182.65:35749)
0928 21:56:45.504110 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 21:56:45.504112 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:56:45.504288 (+ 176us) server_negotiation.cc:407] Connection header received
0928 21:56:45.504359 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:56:45.504359 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:56:45.504385 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:56:45.504452 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:56:45.504609 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:45.505053 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:45.505978 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:45.506414 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:45.506496 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:45.506555 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 21:56:45.506759 (+ 204us) server_negotiation.cc:299] Negotiation successful
0928 21:56:45.506811 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":64,"threads_started":1}
I20260928 21:56:45.506855 17320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b929080d9174670a57c130fef9db996" candidate_uuid: "35f5a5726eea46a988d9b74b3d6e4677" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69a6ebbd4f834658854802f999deed03" is_pre_election: true
I20260928 21:56:45.507000 17320 raft_consensus.cc:2454] T 4b929080d9174670a57c130fef9db996 P 69a6ebbd4f834658854802f999deed03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35f5a5726eea46a988d9b74b3d6e4677 in term 0.
I20260928 21:56:45.507121 17177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b929080d9174670a57c130fef9db996" candidate_uuid: "35f5a5726eea46a988d9b74b3d6e4677" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f91d4c3aa32644a28c325bfc0c55b068" is_pre_election: true
I20260928 21:56:45.507210 17390 leader_election.cc:302] T 4b929080d9174670a57c130fef9db996 P 35f5a5726eea46a988d9b74b3d6e4677 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35f5a5726eea46a988d9b74b3d6e4677, 69a6ebbd4f834658854802f999deed03; no voters:
I20260928 21:56:45.507256 17177 raft_consensus.cc:2454] T 4b929080d9174670a57c130fef9db996 P f91d4c3aa32644a28c325bfc0c55b068 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35f5a5726eea46a988d9b74b3d6e4677 in term 0.
I20260928 21:56:45.507351 17513 raft_consensus.cc:2781] T 4b929080d9174670a57c130fef9db996 P 35f5a5726eea46a988d9b74b3d6e4677 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:56:45.507426 17513 raft_consensus.cc:482] T 4b929080d9174670a57c130fef9db996 P 35f5a5726eea46a988d9b74b3d6e4677 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:56:45.507458 17513 raft_consensus.cc:3037] T 4b929080d9174670a57c130fef9db996 P 35f5a5726eea46a988d9b74b3d6e4677 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:56:45.508188 17513 raft_consensus.cc:504] T 4b929080d9174670a57c130fef9db996 P 35f5a5726eea46a988d9b74b3d6e4677 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f5a5726eea46a988d9b74b3d6e4677" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 41977 } } peers { permanent_uuid: "f91d4c3aa32644a28c325bfc0c55b068" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35749 } } peers { permanent_uuid: "69a6ebbd4f834658854802f999deed03" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 40155 } }
I20260928 21:56:45.508311 17513 leader_election.cc:288] T 4b929080d9174670a57c130fef9db996 P 35f5a5726eea46a988d9b74b3d6e4677 [CANDIDATE]: Term 1 election: Requested vote from peers f91d4c3aa32644a28c325bfc0c55b068 (127.2.182.65:35749), 69a6ebbd4f834658854802f999deed03 (127.2.182.66:40155)
I20260928 21:56:45.508455 17177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b929080d9174670a57c130fef9db996" candidate_uuid: "35f5a5726eea46a988d9b74b3d6e4677" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f91d4c3aa32644a28c325bfc0c55b068"
I20260928 21:56:45.508549 17177 raft_consensus.cc:3037] T 4b929080d9174670a57c130fef9db996 P f91d4c3aa32644a28c325bfc0c55b068 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:56:45.508610 17320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b929080d9174670a57c130fef9db996" candidate_uuid: "35f5a5726eea46a988d9b74b3d6e4677" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69a6ebbd4f834658854802f999deed03"
I20260928 21:56:45.508735 17320 raft_consensus.cc:3037] T 4b929080d9174670a57c130fef9db996 P 69a6ebbd4f834658854802f999deed03 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:56:45.509212 17177 raft_consensus.cc:2454] T 4b929080d9174670a57c130fef9db996 P f91d4c3aa32644a28c325bfc0c55b068 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35f5a5726eea46a988d9b74b3d6e4677 in term 1.
I20260928 21:56:45.509398 17389 leader_election.cc:302] T 4b929080d9174670a57c130fef9db996 P 35f5a5726eea46a988d9b74b3d6e4677 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35f5a5726eea46a988d9b74b3d6e4677, f91d4c3aa32644a28c325bfc0c55b068; no voters:
I20260928 21:56:45.509477 17513 raft_consensus.cc:2781] T 4b929080d9174670a57c130fef9db996 P 35f5a5726eea46a988d9b74b3d6e4677 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:56:45.509577 17320 raft_consensus.cc:2454] T 4b929080d9174670a57c130fef9db996 P 69a6ebbd4f834658854802f999deed03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35f5a5726eea46a988d9b74b3d6e4677 in term 1.
I20260928 21:56:45.509596 17513 raft_consensus.cc:686] T 4b929080d9174670a57c130fef9db996 P 35f5a5726eea46a988d9b74b3d6e4677 [term 1 LEADER]: Becoming Leader. State: Replica: 35f5a5726eea46a988d9b74b3d6e4677, State: Running, Role: LEADER
I20260928 21:56:45.509689 17513 consensus_queue.cc:237] T 4b929080d9174670a57c130fef9db996 P 35f5a5726eea46a988d9b74b3d6e4677 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f5a5726eea46a988d9b74b3d6e4677" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 41977 } } peers { permanent_uuid: "f91d4c3aa32644a28c325bfc0c55b068" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35749 } } peers { permanent_uuid: "69a6ebbd4f834658854802f999deed03" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 40155 } }
I20260928 21:56:45.510432 17039 catalog_manager.cc:5495] T 4b929080d9174670a57c130fef9db996 P 35f5a5726eea46a988d9b74b3d6e4677 reported cstate change: term changed from 0 to 1, leader changed from <none> to 35f5a5726eea46a988d9b74b3d6e4677 (127.2.182.67). New cstate: current_term: 1 leader_uuid: "35f5a5726eea46a988d9b74b3d6e4677" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f5a5726eea46a988d9b74b3d6e4677" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 41977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f91d4c3aa32644a28c325bfc0c55b068" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69a6ebbd4f834658854802f999deed03" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 40155 } health_report { overall_health: UNKNOWN } } }
I20260928 21:56:45.552645 17508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:45.550233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37328 (local address 127.2.182.67:41977)
0928 21:56:45.550322 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0928 21:56:45.550324 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:56:45.550331 (+ 7us) server_negotiation.cc:407] Connection header received
0928 21:56:45.550367 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:56:45.550368 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:56:45.550385 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:56:45.550441 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:56:45.550569 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:45.551049 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:45.551878 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:45.552071 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:45.552124 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:45.552229 (+ 105us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:56:45.552367 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:56:45.552430 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 21:56:45.552521 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 21:56:45.552542 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 21:56:45.554626 17177 raft_consensus.cc:1261] T 4b929080d9174670a57c130fef9db996 P f91d4c3aa32644a28c325bfc0c55b068 [term 1 FOLLOWER]: Refusing update from remote peer 35f5a5726eea46a988d9b74b3d6e4677: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:56:45.554625 17320 raft_consensus.cc:1261] T 4b929080d9174670a57c130fef9db996 P 69a6ebbd4f834658854802f999deed03 [term 1 FOLLOWER]: Refusing update from remote peer 35f5a5726eea46a988d9b74b3d6e4677: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:56:45.554927 17513 consensus_queue.cc:1035] T 4b929080d9174670a57c130fef9db996 P 35f5a5726eea46a988d9b74b3d6e4677 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69a6ebbd4f834658854802f999deed03" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 40155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:56:45.555075 17516 consensus_queue.cc:1035] T 4b929080d9174670a57c130fef9db996 P 35f5a5726eea46a988d9b74b3d6e4677 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f91d4c3aa32644a28c325bfc0c55b068" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:56:45.557065 17519 mvcc.cc:205] Tried to move back new op lower bound from 7334431152348807168 to 7334431152168243200. Current Snapshot: MvccSnapshot[applied={T|T < 7334431152348807168}]
I20260928 21:56:45.557260 17521 mvcc.cc:205] Tried to move back new op lower bound from 7334431152348807168 to 7334431152168243200. Current Snapshot: MvccSnapshot[applied={T|T < 7334431152348807168}]
I20260928 21:56:45.557550 17518 mvcc.cc:205] Tried to move back new op lower bound from 7334431152348807168 to 7334431152168243200. Current Snapshot: MvccSnapshot[applied={T|T < 7334431152348807168}]
I20260928 21:56:45.559653 17039 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:60936:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:56:45.559741 17039 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:60936:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:56:45.560460 17039 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 73350ea0b7a94a888be60798c3d78cb2: Sending DeleteTablet for 3 replicas of tablet 4b929080d9174670a57c130fef9db996
I20260928 21:56:45.560768 17300 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b929080d9174670a57c130fef9db996 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:56:45 UTC) from {username='kudu', principal='kudu/127.2.182.126@KRBTEST.COM'} at 127.0.0.1:50872
I20260928 21:56:45.560796 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 17077
I20260928 21:56:45.560873 17161 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b929080d9174670a57c130fef9db996 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:56:45 UTC) from {username='kudu', principal='kudu/127.2.182.126@KRBTEST.COM'} at 127.0.0.1:35268
I20260928 21:56:45.561244 17373 tablet_replica.cc:330] T 4b929080d9174670a57c130fef9db996 P 69a6ebbd4f834658854802f999deed03: stopping tablet replica
I20260928 21:56:45.561304 17373 raft_consensus.cc:2229] T 4b929080d9174670a57c130fef9db996 P 69a6ebbd4f834658854802f999deed03 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:56:45.561363 17373 raft_consensus.cc:2258] T 4b929080d9174670a57c130fef9db996 P 69a6ebbd4f834658854802f999deed03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:56:45.561703 17373 ts_tablet_manager.cc:1905] T 4b929080d9174670a57c130fef9db996 P 69a6ebbd4f834658854802f999deed03: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:56:45.561691 17436 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b929080d9174670a57c130fef9db996 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:56:45 UTC) from {username='kudu', principal='kudu/127.2.182.126@KRBTEST.COM'} at 127.0.0.1:37326
I20260928 21:56:45.561935 17527 tablet_replica.cc:330] T 4b929080d9174670a57c130fef9db996 P 35f5a5726eea46a988d9b74b3d6e4677: stopping tablet replica
I20260928 21:56:45.562017 17527 raft_consensus.cc:2229] T 4b929080d9174670a57c130fef9db996 P 35f5a5726eea46a988d9b74b3d6e4677 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:56:45.562127 17527 raft_consensus.cc:2258] T 4b929080d9174670a57c130fef9db996 P 35f5a5726eea46a988d9b74b3d6e4677 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:56:45.563477 17373 ts_tablet_manager.cc:1918] T 4b929080d9174670a57c130fef9db996 P 69a6ebbd4f834658854802f999deed03: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:56:45.563534 17373 log.cc:1196] T 4b929080d9174670a57c130fef9db996 P 69a6ebbd4f834658854802f999deed03: Deleting WAL directory at /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-1/wal/wals/4b929080d9174670a57c130fef9db996
I20260928 21:56:45.563596 17527 ts_tablet_manager.cc:1905] T 4b929080d9174670a57c130fef9db996 P 35f5a5726eea46a988d9b74b3d6e4677: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:56:45.563822 17373 ts_tablet_manager.cc:1939] T 4b929080d9174670a57c130fef9db996 P 69a6ebbd4f834658854802f999deed03: Deleting consensus metadata
I20260928 21:56:45.564158 17024 catalog_manager.cc:4841] TS 69a6ebbd4f834658854802f999deed03 (127.2.182.66:40155): tablet 4b929080d9174670a57c130fef9db996 (table test-table [id=44bffbd9e4dd4b4093cad813dfdf9bf6]) successfully deleted
I20260928 21:56:45.565245 17527 ts_tablet_manager.cc:1918] T 4b929080d9174670a57c130fef9db996 P 35f5a5726eea46a988d9b74b3d6e4677: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:56:45.565320 17527 log.cc:1196] T 4b929080d9174670a57c130fef9db996 P 35f5a5726eea46a988d9b74b3d6e4677: Deleting WAL directory at /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-2/wal/wals/4b929080d9174670a57c130fef9db996
I20260928 21:56:45.565567 17527 ts_tablet_manager.cc:1939] T 4b929080d9174670a57c130fef9db996 P 35f5a5726eea46a988d9b74b3d6e4677: Deleting consensus metadata
I20260928 21:56:45.565971 17023 catalog_manager.cc:4841] TS 35f5a5726eea46a988d9b74b3d6e4677 (127.2.182.67:41977): tablet 4b929080d9174670a57c130fef9db996 (table test-table [id=44bffbd9e4dd4b4093cad813dfdf9bf6]) successfully deleted
W20260928 21:56:45.567076 17023 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.182.65:35749 (error 108)
I20260928 21:56:45.567457 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 17232
I20260928 21:56:45.567550 17234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:45.567413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.65:35749 (local address 127.0.0.1:35280)
0928 21:56:45.567470 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:45.567517 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.65:35749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 21:56:45.568046 17023 catalog_manager.cc:4584] TS f91d4c3aa32644a28c325bfc0c55b068 (127.2.182.65:35749): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4b929080d9174670a57c130fef9db996: Network error: Client connection negotiation failed: client connection to 127.2.182.65:35749: connect: Connection refused (error 111)
I20260928 21:56:45.572233 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 17370
I20260928 21:56:45.576426 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 17006
I20260928 21:56:45.581076 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:46141
--webserver_interface=127.2.182.126
--webserver_port=42153
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab
--principal=kudu/127.2.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:33315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.126:46141
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:56:45.660605 17529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:56:45.660833 17529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:56:45.660856 17529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:56:45.662009 17529 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:56:45.662050 17529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:56:45.662063 17529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:56:45.662075 17529 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:56:45.662096 17529 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:56:45.663697 17529 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:33315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.126:46141
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab
--principal=kudu/127.2.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:46141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.126
--webserver_port=42153
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:56:45.663928 17529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:56:45.664141 17529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:56:45.669065 17534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:56:45.669167 17535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:56:45.669065 17537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:56:45.669281 17529 server_base.cc:1034] running on GCE node
I20260928 21:56:45.669576 17529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:56:45.669874 17529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:56:45.671061 17529 hybrid_clock.cc:648] HybridClock initialized: now 1790632605671023 us; error 55 us; skew 500 ppm
Sep 28 21:56:45 dist-test-slave-k6lz krb5kdc[16459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632605, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:56:45.673883 17529 init.cc:377] Logged in from keytab as kudu/127.2.182.126@KRBTEST.COM (short username kudu)
I20260928 21:56:45.674477 17529 webserver.cc:460] Webserver started at http://127.2.182.126:42153/ using document root <none> and password file <none>
I20260928 21:56:45.674707 17529 fs_manager.cc:359] Metadata directory not provided
I20260928 21:56:45.674757 17529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:56:45.676234 17529 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:56:45.676960 17544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:56:45.677209 17529 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:56:45.677296 17529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/master-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/master-0/wal
uuid: "73350ea0b7a94a888be60798c3d78cb2"
format_stamp: "Formatted at 2026-09-28 21:56:43 on dist-test-slave-k6lz"
server_key: "00124e733a592de95d6bf3109f613237"
server_key_iv: "5689f5249293130e727d1f0feb04321f"
server_key_version: "encryptionkey@0"
I20260928 21:56:45.677631 17529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:56:45.704669 17529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:56:45.706779 17529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:56:45.706919 17529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:56:45.710863 17529 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.126:46141
I20260928 21:56:45.710919 17596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.126:46141 every 8 connection(s)
I20260928 21:56:45.711253 17529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
I20260928 21:56:45.712832 17597 sys_catalog.cc:263] Verifying existing consensus state
I20260928 21:56:45.713450 17597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73350ea0b7a94a888be60798c3d78cb2: Bootstrap starting.
I20260928 21:56:45.716050 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 17529
I20260928 21:56:45.716415 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.65:35749
--local_ip_for_outbound_sockets=127.2.182.65
--tserver_master_addrs=127.2.182.126:46141
--webserver_port=37315
--webserver_interface=127.2.182.65
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab
--principal=kudu/127.2.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:33315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:56:45.721554 17597 log.cc:824] T 00000000000000000000000000000000 P 73350ea0b7a94a888be60798c3d78cb2: Log is configured to *not* fsync() on all Append() calls
I20260928 21:56:45.723783 17597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73350ea0b7a94a888be60798c3d78cb2: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 21:56:45.724042 17597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73350ea0b7a94a888be60798c3d78cb2: Bootstrap complete.
I20260928 21:56:45.726150 17597 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73350ea0b7a94a888be60798c3d78cb2 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73350ea0b7a94a888be60798c3d78cb2" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 46141 } }
I20260928 21:56:45.726368 17597 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73350ea0b7a94a888be60798c3d78cb2 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73350ea0b7a94a888be60798c3d78cb2, State: Initialized, Role: FOLLOWER
I20260928 21:56:45.726511 17597 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73350ea0b7a94a888be60798c3d78cb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73350ea0b7a94a888be60798c3d78cb2" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 46141 } }
I20260928 21:56:45.726583 17597 raft_consensus.cc:388] T 00000000000000000000000000000000 P 73350ea0b7a94a888be60798c3d78cb2 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:56:45.726630 17597 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73350ea0b7a94a888be60798c3d78cb2 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:56:45.726677 17597 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73350ea0b7a94a888be60798c3d78cb2 [term 2 FOLLOWER]: Advancing to term 3
I20260928 21:56:45.727682 17597 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73350ea0b7a94a888be60798c3d78cb2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73350ea0b7a94a888be60798c3d78cb2" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 46141 } }
I20260928 21:56:45.727779 17597 leader_election.cc:302] T 00000000000000000000000000000000 P 73350ea0b7a94a888be60798c3d78cb2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73350ea0b7a94a888be60798c3d78cb2; no voters:
I20260928 21:56:45.727934 17597 leader_election.cc:288] T 00000000000000000000000000000000 P 73350ea0b7a94a888be60798c3d78cb2 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 21:56:45.727981 17601 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73350ea0b7a94a888be60798c3d78cb2 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 21:56:45.728096 17601 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73350ea0b7a94a888be60798c3d78cb2 [term 3 LEADER]: Becoming Leader. State: Replica: 73350ea0b7a94a888be60798c3d78cb2, State: Running, Role: LEADER
I20260928 21:56:45.728199 17597 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73350ea0b7a94a888be60798c3d78cb2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:56:45.728186 17601 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73350ea0b7a94a888be60798c3d78cb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73350ea0b7a94a888be60798c3d78cb2" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 46141 } }
I20260928 21:56:45.728456 17601 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73350ea0b7a94a888be60798c3d78cb2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73350ea0b7a94a888be60798c3d78cb2. Latest consensus state: current_term: 3 leader_uuid: "73350ea0b7a94a888be60798c3d78cb2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73350ea0b7a94a888be60798c3d78cb2" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 46141 } } }
I20260928 21:56:45.728518 17601 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73350ea0b7a94a888be60798c3d78cb2 [sys.catalog]: This master's current role is: LEADER
I20260928 21:56:45.728510 17602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73350ea0b7a94a888be60798c3d78cb2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "73350ea0b7a94a888be60798c3d78cb2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73350ea0b7a94a888be60798c3d78cb2" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 46141 } } }
I20260928 21:56:45.728605 17602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73350ea0b7a94a888be60798c3d78cb2 [sys.catalog]: This master's current role is: LEADER
I20260928 21:56:45.728765 17606 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:56:45.729578 17606 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:56:45.729674 17606 catalog_manager.cc:1259] Loaded cluster ID: 71eab75250cc46ec98c3702a895ec83d
I20260928 21:56:45.729709 17606 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260928 21:56:45.729928 17606 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fe0211989ad google::LogMessage::Fail() at ??:0
@ 0x7fe02119b88a google::LogMessage::SendToLog() at ??:0
@ 0x7fe0211984de google::LogMessage::Flush() at ??:0
@ 0x7fe021199a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fe0290b90f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fe0290c28b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fe0230660d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fe02305b954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe023ebf6db start_thread at ??:0
@ 0x7fe01fc9661f clone at ??:0
W20260928 21:56:45.808120 17599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:56:45.808342 17599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:56:45.808380 17599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:56:45.809660 17599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:56:45.809710 17599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:56:45.809747 17599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.65
I20260928 21:56:45.811278 17599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:33315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab
--principal=kudu/127.2.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.65:35749
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.182.65
--webserver_port=37315
--tserver_master_addrs=127.2.182.126:46141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.65
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:56:45.811554 17599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:56:45.811790 17599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:56:45.817078 17634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:56:45.817072 17636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:56:45.817569 17599 server_base.cc:1034] running on GCE node
W20260928 21:56:45.817075 17633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:56:45.817956 17599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:56:45.818274 17599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:56:45.819456 17599 hybrid_clock.cc:648] HybridClock initialized: now 1790632605819436 us; error 36 us; skew 500 ppm
Sep 28 21:56:45 dist-test-slave-k6lz krb5kdc[16459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632605, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:56:45.822606 17599 init.cc:377] Logged in from keytab as kudu/127.2.182.65@KRBTEST.COM (short username kudu)
I20260928 21:56:45.823187 17599 webserver.cc:460] Webserver started at http://127.2.182.65:37315/ using document root <none> and password file <none>
I20260928 21:56:45.823472 17599 fs_manager.cc:359] Metadata directory not provided
I20260928 21:56:45.823540 17599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:56:45.825392 17599 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:56:45.826349 17643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:56:45.826566 17599 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:56:45.826666 17599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-0/wal
uuid: "f91d4c3aa32644a28c325bfc0c55b068"
format_stamp: "Formatted at 2026-09-28 21:56:44 on dist-test-slave-k6lz"
server_key: "10b3104d0bdea2c40b77e84c40c1f9e1"
server_key_iv: "3595306e0995fcca1342728259a51ad5"
server_key_version: "encryptionkey@0"
I20260928 21:56:45.827128 17599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:56:45.847029 17599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:56:45.849802 17599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:56:45.849978 17599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:56:45.850267 17599 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:56:45.850813 17650 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 21:56:45.852401 17599 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 21:56:45.852461 17599 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 21:56:45.852496 17599 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 21:56:45.853386 17599 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 21:56:45.853432 17599 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 21:56:45.853509 17650 tablet_bootstrap.cc:492] T 4b929080d9174670a57c130fef9db996 P f91d4c3aa32644a28c325bfc0c55b068: Bootstrap starting.
I20260928 21:56:45.860925 17599 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.65:35749
I20260928 21:56:45.861565 17599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
I20260928 21:56:45.861752 17757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.65:35749 every 8 connection(s)
I20260928 21:56:45.864262 17650 log.cc:824] T 4b929080d9174670a57c130fef9db996 P f91d4c3aa32644a28c325bfc0c55b068: Log is configured to *not* fsync() on all Append() calls
I20260928 21:56:45.866381 17650 tablet_bootstrap.cc:492] T 4b929080d9174670a57c130fef9db996 P f91d4c3aa32644a28c325bfc0c55b068: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 21:56:45.866657 17650 tablet_bootstrap.cc:492] T 4b929080d9174670a57c130fef9db996 P f91d4c3aa32644a28c325bfc0c55b068: Bootstrap complete.
I20260928 21:56:45.867635 17650 ts_tablet_manager.cc:1397] T 4b929080d9174670a57c130fef9db996 P f91d4c3aa32644a28c325bfc0c55b068: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.011s
I20260928 21:56:45.868706 17650 raft_consensus.cc:348] T 4b929080d9174670a57c130fef9db996 P f91d4c3aa32644a28c325bfc0c55b068 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f5a5726eea46a988d9b74b3d6e4677" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 41977 } } peers { permanent_uuid: "f91d4c3aa32644a28c325bfc0c55b068" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35749 } } peers { permanent_uuid: "69a6ebbd4f834658854802f999deed03" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 40155 } }
I20260928 21:56:45.868943 17650 raft_consensus.cc:729] T 4b929080d9174670a57c130fef9db996 P f91d4c3aa32644a28c325bfc0c55b068 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f91d4c3aa32644a28c325bfc0c55b068, State: Initialized, Role: FOLLOWER
I20260928 21:56:45.869073 17650 consensus_queue.cc:260] T 4b929080d9174670a57c130fef9db996 P f91d4c3aa32644a28c325bfc0c55b068 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f5a5726eea46a988d9b74b3d6e4677" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 41977 } } peers { permanent_uuid: "f91d4c3aa32644a28c325bfc0c55b068" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35749 } } peers { permanent_uuid: "69a6ebbd4f834658854802f999deed03" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 40155 } }
I20260928 21:56:45.869360 17650 ts_tablet_manager.cc:1428] T 4b929080d9174670a57c130fef9db996 P f91d4c3aa32644a28c325bfc0c55b068: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 21:56:45.871730 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 17599
I20260928 21:56:45.871961 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.66:40155
--local_ip_for_outbound_sockets=127.2.182.66
--tserver_master_addrs=127.2.182.126:46141
--webserver_port=45681
--webserver_interface=127.2.182.66
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab
--principal=kudu/127.2.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:33315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:56:45.940858 17760 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:56:45.862593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:46141 (local address 127.2.182.65:33061)
0928 21:56:45.862826 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:45.862877 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 21:56:45.863577 (+ 700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:56:45.940649 (+ 77072us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:46141: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":117,"threads_started":1}
W20260928 21:56:45.941112 17758 heartbeater.cc:644] Failed to heartbeat to 127.2.182.126:46141 (0 consecutive failures): Network error: Failed to ping master at 127.2.182.126:46141: Client connection negotiation failed: client connection to 127.2.182.126:46141: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 21:56:45.941514 17760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:45.941391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:46141 (local address 127.2.182.65:59181)
0928 21:56:45.941441 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:45.941488 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:46141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 21:56:45.941862 17760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:45.941773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:46141 (local address 127.2.182.65:41235)
0928 21:56:45.941814 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:45.941841 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:46141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 21:56:45.941974 17758 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 21:56:45.955835 17763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:56:45.956048 17763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:56:45.956071 17763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:56:45.957329 17763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:56:45.957377 17763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:56:45.957404 17763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.66
I20260928 21:56:45.958932 17763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:33315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab
--principal=kudu/127.2.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.66:40155
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.182.66
--webserver_port=45681
--tserver_master_addrs=127.2.182.126:46141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.66
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:56:45.959158 17763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:56:45.959439 17763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:56:45.964682 17768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:56:45.964689 17769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:56:45.964685 17771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:56:45.964774 17763 server_base.cc:1034] running on GCE node
I20260928 21:56:45.965065 17763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:56:45.965301 17763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:56:45.966481 17763 hybrid_clock.cc:648] HybridClock initialized: now 1790632605966459 us; error 34 us; skew 500 ppm
Sep 28 21:56:45 dist-test-slave-k6lz krb5kdc[16459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632605, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:56:45.969072 17763 init.cc:377] Logged in from keytab as kudu/127.2.182.66@KRBTEST.COM (short username kudu)
I20260928 21:56:45.969612 17763 webserver.cc:460] Webserver started at http://127.2.182.66:45681/ using document root <none> and password file <none>
I20260928 21:56:45.969831 17763 fs_manager.cc:359] Metadata directory not provided
I20260928 21:56:45.969875 17763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:56:45.971267 17763 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:56:45.971951 17778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:56:45.972226 17763 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 21:56:45.972311 17763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-1/wal
uuid: "69a6ebbd4f834658854802f999deed03"
format_stamp: "Formatted at 2026-09-28 21:56:44 on dist-test-slave-k6lz"
server_key: "e7d421bfce3f3143d1eb912196f2d41d"
server_key_iv: "711d2ac6a7ace06e7156feee47c58646"
server_key_version: "encryptionkey@0"
I20260928 21:56:45.972679 17763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:56:45.993878 17763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:56:45.996019 17763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:56:45.996162 17763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:56:45.996384 17763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:56:45.996665 17763 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:56:45.996697 17763 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:56:45.996724 17763 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:56:45.996747 17763 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:56:46.002475 17763 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.66:40155
I20260928 21:56:46.002516 17891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.66:40155 every 8 connection(s)
I20260928 21:56:46.002856 17763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
I20260928 21:56:46.004046 17894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:46.003572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:46141 (local address 127.2.182.66:39915)
0928 21:56:46.003819 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:46.003895 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:46141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":74,"threads_started":1}
W20260928 21:56:46.004550 17892 heartbeater.cc:644] Failed to heartbeat to 127.2.182.126:46141 (0 consecutive failures): Network error: Failed to ping master at 127.2.182.126:46141: Client connection negotiation failed: client connection to 127.2.182.126:46141: connect: Connection refused (error 111)
I20260928 21:56:46.004875 17894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:46.004757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:46141 (local address 127.2.182.66:34685)
0928 21:56:46.004817 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:46.004847 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:46141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 21:56:46.005235 17894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:46.005086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:46141 (local address 127.2.182.66:36131)
0928 21:56:46.005186 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:46.005219 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:46141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260928 21:56:46.005442 17892 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 21:56:46.007561 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 17763
I20260928 21:56:46.007812 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.67:41977
--local_ip_for_outbound_sockets=127.2.182.67
--tserver_master_addrs=127.2.182.126:46141
--webserver_port=36599
--webserver_interface=127.2.182.67
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab
--principal=kudu/127.2.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:33315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:56:46.089574 17895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:56:46.089808 17895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:56:46.089836 17895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:56:46.091037 17895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:56:46.091081 17895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:56:46.091109 17895 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.67
I20260928 21:56:46.092679 17895 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:33315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab
--principal=kudu/127.2.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.67:41977
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.182.67
--webserver_port=36599
--tserver_master_addrs=127.2.182.126:46141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17895
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.67
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:56:46.092916 17895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:56:46.093128 17895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:56:46.098135 17900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:56:46.098137 17901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:56:46.098325 17895 server_base.cc:1034] running on GCE node
W20260928 21:56:46.098135 17903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:56:46.098649 17895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:56:46.098891 17895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:56:46.100019 17895 hybrid_clock.cc:648] HybridClock initialized: now 1790632606100009 us; error 28 us; skew 500 ppm
Sep 28 21:56:46 dist-test-slave-k6lz krb5kdc[16459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632606, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:56:46.102450 17895 init.cc:377] Logged in from keytab as kudu/127.2.182.67@KRBTEST.COM (short username kudu)
I20260928 21:56:46.102890 17895 webserver.cc:460] Webserver started at http://127.2.182.67:36599/ using document root <none> and password file <none>
I20260928 21:56:46.103068 17895 fs_manager.cc:359] Metadata directory not provided
I20260928 21:56:46.103106 17895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:56:46.104476 17895 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:56:46.105204 17910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:56:46.105409 17895 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:56:46.105499 17895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-2/wal
uuid: "35f5a5726eea46a988d9b74b3d6e4677"
format_stamp: "Formatted at 2026-09-28 21:56:44 on dist-test-slave-k6lz"
server_key: "91ad26c273c29d765986c82a4b03fcda"
server_key_iv: "8c48c0813c725fb0cdeb3e3a2679313e"
server_key_version: "encryptionkey@0"
I20260928 21:56:46.105846 17895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:56:46.121338 17895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:56:46.123715 17895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:56:46.123833 17895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:56:46.124078 17895 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:56:46.124358 17895 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:56:46.124388 17895 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:56:46.124410 17895 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:56:46.124426 17895 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:56:46.130111 17895 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.67:41977
I20260928 21:56:46.130165 18023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.67:41977 every 8 connection(s)
I20260928 21:56:46.130503 17895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
I20260928 21:56:46.131650 18026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:46.131214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:46141 (local address 127.2.182.67:40487)
0928 21:56:46.131407 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:46.131486 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:46141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":80,"threads_started":1}
W20260928 21:56:46.132086 18024 heartbeater.cc:644] Failed to heartbeat to 127.2.182.126:46141 (0 consecutive failures): Network error: Failed to ping master at 127.2.182.126:46141: Client connection negotiation failed: client connection to 127.2.182.126:46141: connect: Connection refused (error 111)
I20260928 21:56:46.132448 18026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:46.132328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:46141 (local address 127.2.182.67:59923)
0928 21:56:46.132393 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:46.132421 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:46141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 21:56:46.132799 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 17895
I20260928 21:56:46.132835 18026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:46.132737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:46141 (local address 127.2.182.67:56413)
0928 21:56:46.132787 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:46.132815 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:46141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 21:56:46.132951 18024 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 21:56:46.942576 18028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:46.942346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:46141 (local address 127.2.182.65:46893)
0928 21:56:46.942474 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:46.942516 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:46141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":64,"threads_started":1}
I20260928 21:56:47.006047 18029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:47.005809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:46141 (local address 127.2.182.66:60393)
0928 21:56:47.005947 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:47.005990 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:46141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":58,"threads_started":1}
I20260928 21:56:47.133522 18030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:47.133296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:46141 (local address 127.2.182.67:43669)
0928 21:56:47.133424 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:47.133466 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:46141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":58,"threads_started":1}
I20260928 21:56:47.137639 18031 raft_consensus.cc:482] T 4b929080d9174670a57c130fef9db996 P f91d4c3aa32644a28c325bfc0c55b068 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:56:47.137755 18031 raft_consensus.cc:504] T 4b929080d9174670a57c130fef9db996 P f91d4c3aa32644a28c325bfc0c55b068 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f5a5726eea46a988d9b74b3d6e4677" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 41977 } } peers { permanent_uuid: "f91d4c3aa32644a28c325bfc0c55b068" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35749 } } peers { permanent_uuid: "69a6ebbd4f834658854802f999deed03" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 40155 } }
I20260928 21:56:47.138075 18031 leader_election.cc:288] T 4b929080d9174670a57c130fef9db996 P f91d4c3aa32644a28c325bfc0c55b068 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 35f5a5726eea46a988d9b74b3d6e4677 (127.2.182.67:41977), 69a6ebbd4f834658854802f999deed03 (127.2.182.66:40155)
Sep 28 21:56:47 dist-test-slave-k6lz krb5kdc[16459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632605, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.65@KRBTEST.COM for kudu/127.2.182.66@KRBTEST.COM
Sep 28 21:56:47 dist-test-slave-k6lz krb5kdc[16459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632605, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.65@KRBTEST.COM for kudu/127.2.182.67@KRBTEST.COM
I20260928 21:56:47.147806 18034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:47.138347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.65:56457 (local address 127.2.182.66:40155)
0928 21:56:47.138464 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 21:56:47.138474 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 21:56:47.138504 (+ 30us) server_negotiation.cc:407] Connection header received
0928 21:56:47.138920 (+ 416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:56:47.138922 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:56:47.138958 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:56:47.139078 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:56:47.139821 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:47.140665 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:47.141671 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:47.142080 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:47.142174 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:47.144398 (+ 2224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:56:47.144403 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:56:47.144403 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:56:47.144415 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:56:47.146126 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:56:47.146583 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:56:47.146588 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:56:47.146588 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:56:47.146682 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:56:47.146923 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:56:47.146924 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:56:47.146924 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:56:47.147162 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:56:47.147237 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 21:56:47.147573 (+ 336us) server_negotiation.cc:299] Negotiation successful
0928 21:56:47.147693 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":55,"threads_started":1}
I20260928 21:56:47.147814 18032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:47.138289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.65:51641 (local address 127.2.182.67:41977)
0928 21:56:47.138429 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 21:56:47.138440 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 21:56:47.138473 (+ 33us) server_negotiation.cc:407] Connection header received
0928 21:56:47.138916 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:56:47.138918 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:56:47.138958 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:56:47.139084 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:56:47.139822 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:47.140666 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:47.141673 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:56:47.142080 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:56:47.142174 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:47.144650 (+ 2476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:56:47.144653 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:56:47.144653 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:56:47.144663 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:56:47.146230 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:56:47.146601 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:56:47.146603 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:56:47.146604 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:56:47.146684 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:56:47.146922 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:56:47.146923 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:56:47.146923 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:56:47.147145 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:56:47.147207 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 21:56:47.147555 (+ 348us) server_negotiation.cc:299] Negotiation successful
0928 21:56:47.147674 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":52,"threads_started":1}
I20260928 21:56:47.147827 18028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:47.138213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.67:41977 (local address 127.2.182.65:51641)
0928 21:56:47.138260 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:47.138272 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:56:47.138329 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:56:47.139138 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:56:47.139141 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:56:47.139519 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:56:47.139712 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:56:47.139717 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:56:47.140777 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:56:47.140778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:56:47.141594 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:56:47.141596 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:56:47.142160 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:56:47.142161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:56:47.142272 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:47.142796 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:56:47.142803 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:56:47.144542 (+ 1739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:56:47.146317 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:56:47.146318 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:56:47.146318 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:56:47.146514 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:56:47.146745 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:56:47.146745 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:56:47.146746 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:56:47.146861 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:56:47.147233 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:56:47.147239 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:56:47.147395 (+ 156us) client_negotiation.cc:769] Sending connection context
0928 21:56:47.147494 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 21:56:47.147701 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":61}
I20260928 21:56:47.148423 18033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:47.138219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.66:40155 (local address 127.2.182.65:56457)
0928 21:56:47.138432 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:47.138441 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 21:56:47.138480 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:56:47.139138 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:56:47.139141 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:56:47.139518 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:56:47.139712 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:56:47.139717 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:56:47.140765 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:56:47.140766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:56:47.141593 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:56:47.141595 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:56:47.142162 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:56:47.142163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:56:47.142275 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:56:47.142824 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:56:47.142829 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:56:47.144250 (+ 1421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:56:47.146212 (+ 1962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:56:47.146219 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:56:47.146221 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:56:47.146489 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:56:47.146746 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:56:47.146747 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:56:47.146747 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:56:47.146861 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:56:47.147231 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:56:47.147307 (+ 76us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:56:47.147395 (+ 88us) client_negotiation.cc:769] Sending connection context
0928 21:56:47.147494 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 21:56:47.147775 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":267,"thread_start_us":56,"threads_started":1}
W20260928 21:56:47.148841 17644 leader_election.cc:341] T 4b929080d9174670a57c130fef9db996 P f91d4c3aa32644a28c325bfc0c55b068 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 35f5a5726eea46a988d9b74b3d6e4677 (127.2.182.67:41977): Not found: Tablet not found: 4b929080d9174670a57c130fef9db996
W20260928 21:56:47.149106 17645 leader_election.cc:341] T 4b929080d9174670a57c130fef9db996 P f91d4c3aa32644a28c325bfc0c55b068 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 69a6ebbd4f834658854802f999deed03 (127.2.182.66:40155): Not found: Tablet not found: 4b929080d9174670a57c130fef9db996
I20260928 21:56:47.149159 17645 leader_election.cc:302] T 4b929080d9174670a57c130fef9db996 P f91d4c3aa32644a28c325bfc0c55b068 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f91d4c3aa32644a28c325bfc0c55b068; no voters: 35f5a5726eea46a988d9b74b3d6e4677, 69a6ebbd4f834658854802f999deed03
I20260928 21:56:47.149326 18031 raft_consensus.cc:2726] T 4b929080d9174670a57c130fef9db996 P f91d4c3aa32644a28c325bfc0c55b068 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 21:56:47.943312 18035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:47.943040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:46141 (local address 127.2.182.65:60775)
0928 21:56:47.943230 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:47.943267 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:46141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":78,"threads_started":1}
I20260928 21:56:48.006789 18036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:48.006539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:46141 (local address 127.2.182.66:35961)
0928 21:56:48.006704 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:48.006746 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:46141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":65,"threads_started":1}
I20260928 21:56:48.134378 18037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:48.134047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:46141 (local address 127.2.182.67:44065)
0928 21:56:48.134293 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:48.134333 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:46141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":177,"threads_started":1}
I20260928 21:56:48.847961 18038 raft_consensus.cc:482] T 4b929080d9174670a57c130fef9db996 P f91d4c3aa32644a28c325bfc0c55b068 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:56:48.848060 18038 raft_consensus.cc:504] T 4b929080d9174670a57c130fef9db996 P f91d4c3aa32644a28c325bfc0c55b068 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f5a5726eea46a988d9b74b3d6e4677" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 41977 } } peers { permanent_uuid: "f91d4c3aa32644a28c325bfc0c55b068" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35749 } } peers { permanent_uuid: "69a6ebbd4f834658854802f999deed03" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 40155 } }
I20260928 21:56:48.848230 18038 leader_election.cc:288] T 4b929080d9174670a57c130fef9db996 P f91d4c3aa32644a28c325bfc0c55b068 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 35f5a5726eea46a988d9b74b3d6e4677 (127.2.182.67:41977), 69a6ebbd4f834658854802f999deed03 (127.2.182.66:40155)
W20260928 21:56:48.848615 17645 leader_election.cc:341] T 4b929080d9174670a57c130fef9db996 P f91d4c3aa32644a28c325bfc0c55b068 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 69a6ebbd4f834658854802f999deed03 (127.2.182.66:40155): Not found: Tablet not found: 4b929080d9174670a57c130fef9db996
W20260928 21:56:48.848675 17644 leader_election.cc:341] T 4b929080d9174670a57c130fef9db996 P f91d4c3aa32644a28c325bfc0c55b068 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 35f5a5726eea46a988d9b74b3d6e4677 (127.2.182.67:41977): Not found: Tablet not found: 4b929080d9174670a57c130fef9db996
I20260928 21:56:48.848752 17644 leader_election.cc:302] T 4b929080d9174670a57c130fef9db996 P f91d4c3aa32644a28c325bfc0c55b068 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f91d4c3aa32644a28c325bfc0c55b068; no voters: 35f5a5726eea46a988d9b74b3d6e4677, 69a6ebbd4f834658854802f999deed03
I20260928 21:56:48.848843 18038 raft_consensus.cc:2726] T 4b929080d9174670a57c130fef9db996 P f91d4c3aa32644a28c325bfc0c55b068 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 21:56:48.944026 18039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:48.943814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:46141 (local address 127.2.182.65:52331)
0928 21:56:48.943933 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:48.943964 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:46141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":64,"threads_started":1}
I20260928 21:56:49.007616 18040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:49.007398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:46141 (local address 127.2.182.66:54629)
0928 21:56:49.007533 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:49.007569 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:46141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":61,"threads_started":1}
I20260928 21:56:49.135162 18041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:49.134915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:46141 (local address 127.2.182.67:40343)
0928 21:56:49.135062 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:49.135116 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:46141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":65,"threads_started":1}
I20260928 21:56:49.944840 18042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:49.944542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:46141 (local address 127.2.182.65:41289)
0928 21:56:49.944741 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:49.944793 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:46141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":93,"threads_started":1}
I20260928 21:56:50.008474 18043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:50.008212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:46141 (local address 127.2.182.66:35413)
0928 21:56:50.008375 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:50.008425 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:46141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":73,"threads_started":1}
I20260928 21:56:50.135987 18044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:50.135762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:46141 (local address 127.2.182.67:50025)
0928 21:56:50.135898 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:50.135940 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:46141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":76,"threads_started":1}
I20260928 21:56:50.622130 18045 raft_consensus.cc:482] T 4b929080d9174670a57c130fef9db996 P f91d4c3aa32644a28c325bfc0c55b068 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:56:50.622231 18045 raft_consensus.cc:504] T 4b929080d9174670a57c130fef9db996 P f91d4c3aa32644a28c325bfc0c55b068 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f5a5726eea46a988d9b74b3d6e4677" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 41977 } } peers { permanent_uuid: "f91d4c3aa32644a28c325bfc0c55b068" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35749 } } peers { permanent_uuid: "69a6ebbd4f834658854802f999deed03" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 40155 } }
I20260928 21:56:50.622380 18045 leader_election.cc:288] T 4b929080d9174670a57c130fef9db996 P f91d4c3aa32644a28c325bfc0c55b068 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 35f5a5726eea46a988d9b74b3d6e4677 (127.2.182.67:41977), 69a6ebbd4f834658854802f999deed03 (127.2.182.66:40155)
W20260928 21:56:50.622705 17644 leader_election.cc:341] T 4b929080d9174670a57c130fef9db996 P f91d4c3aa32644a28c325bfc0c55b068 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 35f5a5726eea46a988d9b74b3d6e4677 (127.2.182.67:41977): Not found: Tablet not found: 4b929080d9174670a57c130fef9db996
W20260928 21:56:50.622843 17645 leader_election.cc:341] T 4b929080d9174670a57c130fef9db996 P f91d4c3aa32644a28c325bfc0c55b068 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 69a6ebbd4f834658854802f999deed03 (127.2.182.66:40155): Not found: Tablet not found: 4b929080d9174670a57c130fef9db996
I20260928 21:56:50.622905 17645 leader_election.cc:302] T 4b929080d9174670a57c130fef9db996 P f91d4c3aa32644a28c325bfc0c55b068 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f91d4c3aa32644a28c325bfc0c55b068; no voters: 35f5a5726eea46a988d9b74b3d6e4677, 69a6ebbd4f834658854802f999deed03
I20260928 21:56:50.622970 18045 raft_consensus.cc:2726] T 4b929080d9174670a57c130fef9db996 P f91d4c3aa32644a28c325bfc0c55b068 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 21:56:50.945755 18046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:50.945417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:46141 (local address 127.2.182.65:53849)
0928 21:56:50.945611 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:50.945658 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:46141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":81,"threads_started":1}
I20260928 21:56:51.009315 18047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:51.009024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:46141 (local address 127.2.182.66:53107)
0928 21:56:51.009215 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:51.009264 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:46141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":84,"threads_started":1}
I20260928 21:56:51.136745 18048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:51.136536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:46141 (local address 127.2.182.67:52115)
0928 21:56:51.136659 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:51.136701 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:46141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":62,"threads_started":1}
I20260928 21:56:51.946625 18049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:51.946344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:46141 (local address 127.2.182.65:54625)
0928 21:56:51.946521 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:51.946576 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:46141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":73,"threads_started":1}
I20260928 21:56:52.010231 18050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:52.009938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:46141 (local address 127.2.182.66:41871)
0928 21:56:52.010117 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:52.010172 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:46141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":76,"threads_started":1}
I20260928 21:56:52.137390 18051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:52.137150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:46141 (local address 127.2.182.67:59315)
0928 21:56:52.137295 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:52.137337 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:46141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":71,"threads_started":1}
I20260928 21:56:52.947398 18052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:52.947102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:46141 (local address 127.2.182.65:53767)
0928 21:56:52.947266 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:52.947310 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:46141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":63,"threads_started":1}
I20260928 21:56:53.010942 18053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:53.010687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:46141 (local address 127.2.182.66:33555)
0928 21:56:53.010848 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:53.010890 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:46141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":78,"threads_started":1}
I20260928 21:56:53.138042 18054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:53.137827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:46141 (local address 127.2.182.67:54717)
0928 21:56:53.137958 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:53.137997 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:46141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":60,"threads_started":1}
I20260928 21:56:53.948137 18055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:53.947909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:46141 (local address 127.2.182.65:58655)
0928 21:56:53.948042 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:53.948086 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:46141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":65,"threads_started":1}
I20260928 21:56:54.011837 18056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:54.011557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:46141 (local address 127.2.182.66:49849)
0928 21:56:54.011742 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:54.011788 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:46141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":64,"threads_started":1}
I20260928 21:56:54.138881 18057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:54.138608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:46141 (local address 127.2.182.67:40061)
0928 21:56:54.138790 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:54.138834 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:46141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":77,"threads_started":1}
I20260928 21:56:54.887193 18058 raft_consensus.cc:482] T 4b929080d9174670a57c130fef9db996 P f91d4c3aa32644a28c325bfc0c55b068 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:56:54.887301 18058 raft_consensus.cc:504] T 4b929080d9174670a57c130fef9db996 P f91d4c3aa32644a28c325bfc0c55b068 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f5a5726eea46a988d9b74b3d6e4677" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 41977 } } peers { permanent_uuid: "f91d4c3aa32644a28c325bfc0c55b068" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35749 } } peers { permanent_uuid: "69a6ebbd4f834658854802f999deed03" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 40155 } }
I20260928 21:56:54.887517 18058 leader_election.cc:288] T 4b929080d9174670a57c130fef9db996 P f91d4c3aa32644a28c325bfc0c55b068 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 35f5a5726eea46a988d9b74b3d6e4677 (127.2.182.67:41977), 69a6ebbd4f834658854802f999deed03 (127.2.182.66:40155)
W20260928 21:56:54.887931 17644 leader_election.cc:341] T 4b929080d9174670a57c130fef9db996 P f91d4c3aa32644a28c325bfc0c55b068 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 35f5a5726eea46a988d9b74b3d6e4677 (127.2.182.67:41977): Not found: Tablet not found: 4b929080d9174670a57c130fef9db996
W20260928 21:56:54.888008 17645 leader_election.cc:341] T 4b929080d9174670a57c130fef9db996 P f91d4c3aa32644a28c325bfc0c55b068 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 69a6ebbd4f834658854802f999deed03 (127.2.182.66:40155): Not found: Tablet not found: 4b929080d9174670a57c130fef9db996
I20260928 21:56:54.888063 17645 leader_election.cc:302] T 4b929080d9174670a57c130fef9db996 P f91d4c3aa32644a28c325bfc0c55b068 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f91d4c3aa32644a28c325bfc0c55b068; no voters: 35f5a5726eea46a988d9b74b3d6e4677, 69a6ebbd4f834658854802f999deed03
I20260928 21:56:54.888149 18058 raft_consensus.cc:2726] T 4b929080d9174670a57c130fef9db996 P f91d4c3aa32644a28c325bfc0c55b068 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 21:56:54.949101 18059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:54.948861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:46141 (local address 127.2.182.65:53911)
0928 21:56:54.949019 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:54.949054 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:46141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":70,"threads_started":1}
I20260928 21:56:55.012706 18060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:55.012424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:46141 (local address 127.2.182.66:33391)
0928 21:56:55.012593 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:55.012647 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:46141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":87,"threads_started":1}
I20260928 21:56:55.139693 18061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:55.139428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:46141 (local address 127.2.182.67:34209)
0928 21:56:55.139580 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:55.139631 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:46141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":58,"threads_started":1}
I20260928 21:56:55.949947 18062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:55.949658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:46141 (local address 127.2.182.65:48981)
0928 21:56:55.949850 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:55.949898 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:46141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":71,"threads_started":1}
I20260928 21:56:56.013554 18063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:56.013294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:46141 (local address 127.2.182.66:37881)
0928 21:56:56.013451 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:56.013502 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:46141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":66,"threads_started":1}
I20260928 21:56:56.140421 18064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:56.140161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:46141 (local address 127.2.182.67:50333)
0928 21:56:56.140326 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:56.140373 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:46141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":63,"threads_started":1}
I20260928 21:56:56.950771 18065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:56.950541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:46141 (local address 127.2.182.65:53537)
0928 21:56:56.950662 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:56.950701 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:46141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":61,"threads_started":1}
I20260928 21:56:57.014353 18066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:57.014080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:46141 (local address 127.2.182.66:46957)
0928 21:56:57.014252 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:57.014300 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:46141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":74,"threads_started":1}
I20260928 21:56:57.141089 18067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:57.140874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:46141 (local address 127.2.182.67:32859)
0928 21:56:57.140999 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:57.141041 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:46141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":61,"threads_started":1}
I20260928 21:56:57.951581 18068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:57.951324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:46141 (local address 127.2.182.65:58989)
0928 21:56:57.951484 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:57.951530 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:46141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":69,"threads_started":1}
I20260928 21:56:58.015151 18069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:58.014936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:46141 (local address 127.2.182.66:47077)
0928 21:56:58.015060 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:58.015103 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:46141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":68,"threads_started":1}
I20260928 21:56:58.141813 18070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:58.141555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:46141 (local address 127.2.182.67:42353)
0928 21:56:58.141716 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:58.141763 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:46141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":77,"threads_started":1}
I20260928 21:56:58.832389 18071 raft_consensus.cc:482] T 4b929080d9174670a57c130fef9db996 P f91d4c3aa32644a28c325bfc0c55b068 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:56:58.832499 18071 raft_consensus.cc:504] T 4b929080d9174670a57c130fef9db996 P f91d4c3aa32644a28c325bfc0c55b068 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f5a5726eea46a988d9b74b3d6e4677" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 41977 } } peers { permanent_uuid: "f91d4c3aa32644a28c325bfc0c55b068" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 35749 } } peers { permanent_uuid: "69a6ebbd4f834658854802f999deed03" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 40155 } }
I20260928 21:56:58.832693 18071 leader_election.cc:288] T 4b929080d9174670a57c130fef9db996 P f91d4c3aa32644a28c325bfc0c55b068 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 35f5a5726eea46a988d9b74b3d6e4677 (127.2.182.67:41977), 69a6ebbd4f834658854802f999deed03 (127.2.182.66:40155)
W20260928 21:56:58.833058 17644 leader_election.cc:341] T 4b929080d9174670a57c130fef9db996 P f91d4c3aa32644a28c325bfc0c55b068 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 35f5a5726eea46a988d9b74b3d6e4677 (127.2.182.67:41977): Not found: Tablet not found: 4b929080d9174670a57c130fef9db996
W20260928 21:56:58.833158 17645 leader_election.cc:341] T 4b929080d9174670a57c130fef9db996 P f91d4c3aa32644a28c325bfc0c55b068 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 69a6ebbd4f834658854802f999deed03 (127.2.182.66:40155): Not found: Tablet not found: 4b929080d9174670a57c130fef9db996
I20260928 21:56:58.833220 17645 leader_election.cc:302] T 4b929080d9174670a57c130fef9db996 P f91d4c3aa32644a28c325bfc0c55b068 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f91d4c3aa32644a28c325bfc0c55b068; no voters: 35f5a5726eea46a988d9b74b3d6e4677, 69a6ebbd4f834658854802f999deed03
I20260928 21:56:58.833299 18071 raft_consensus.cc:2726] T 4b929080d9174670a57c130fef9db996 P f91d4c3aa32644a28c325bfc0c55b068 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 21:56:58.952355 18072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:58.952091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:46141 (local address 127.2.182.65:34163)
0928 21:56:58.952254 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:58.952298 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:46141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":67,"threads_started":1}
I20260928 21:56:59.015882 18073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:59.015633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:46141 (local address 127.2.182.66:50033)
0928 21:56:59.015793 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:59.015836 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:46141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":84,"threads_started":1}
I20260928 21:56:59.142556 18074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:59.142327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:46141 (local address 127.2.182.67:44447)
0928 21:56:59.142469 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:59.142510 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:46141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":87,"threads_started":1}
I20260928 21:56:59.953253 18075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:56:59.952922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:46141 (local address 127.2.182.65:39025)
0928 21:56:59.953091 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 21:56:59.953138 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:46141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":72,"threads_started":1}
I20260928 21:57:00.016659 18076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:00.016391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:46141 (local address 127.2.182.66:44179)
0928 21:57:00.016538 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:00.016594 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:46141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":72,"threads_started":1}
I20260928 21:57:00.143321 18077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:00.143032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:46141 (local address 127.2.182.67:40059)
0928 21:57:00.143224 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:00.143271 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:46141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":79,"threads_started":1}
I20260928 21:57:00.954103 18078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:00.953889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:46141 (local address 127.2.182.65:33189)
0928 21:57:00.954020 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:00.954056 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:46141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":63,"threads_started":1}
I20260928 21:57:01.017442 18079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:01.017184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:46141 (local address 127.2.182.66:38561)
0928 21:57:01.017351 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:01.017388 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:46141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":71,"threads_started":1}
I20260928 21:57:01.143180 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 17599
I20260928 21:57:01.144793 18080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:01.144422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:46141 (local address 127.2.182.67:36765)
0928 21:57:01.144693 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:01.144742 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:46141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":60,"threads_started":1}
I20260928 21:57:01.147540 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 17763
I20260928 21:57:01.151312 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 17895
2026-09-28T21:57:01Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18387 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:57:01 dist-test-slave-k6lz krb5kdc[18083](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:57:01 dist-test-slave-k6lz krb5kdc[18083](info): set up 2 sockets
Sep 28 21:57:01 dist-test-slave-k6lz krb5kdc[18083](info): commencing operation
krb5kdc: starting...
W20260928 21:57:02.115257 2777 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Sep 28 21:57:02 dist-test-slave-k6lz krb5kdc[18083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:57:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:57:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab.
Entry for principal kudu/127.2.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab.
I20260928 21:57:02.236048 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:42609
--webserver_interface=127.2.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab
--principal=kudu/127.2.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:41569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.126:42609
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:57:02.318754 18096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:57:02.318962 18096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:57:02.318984 18096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:57:02.320207 18096 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 21:57:02.320251 18096 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 21:57:02.320263 18096 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 21:57:02.320273 18096 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:57:02.320284 18096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:57:02.320297 18096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:57:02.320307 18096 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:57:02.320317 18096 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:57:02.321808 18096 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:41569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.182.126:42609
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab
--principal=kudu/127.2.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:42609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:57:02.322007 18096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:57:02.322182 18096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:57:02.327071 18101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:57:02.327065 18104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:57:02.327065 18102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:57:02.327226 18096 server_base.cc:1034] running on GCE node
I20260928 21:57:02.327567 18096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:57:02.327854 18096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:57:02.329015 18096 hybrid_clock.cc:648] HybridClock initialized: now 1790632622328994 us; error 36 us; skew 500 ppm
Sep 28 21:57:02 dist-test-slave-k6lz krb5kdc[18083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632622, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:57:02.331781 18096 init.cc:377] Logged in from keytab as kudu/127.2.182.126@KRBTEST.COM (short username kudu)
I20260928 21:57:02.332230 18096 webserver.cc:460] Webserver started at http://127.2.182.126:46749/ using document root <none> and password file <none>
I20260928 21:57:02.332513 18096 fs_manager.cc:359] Metadata directory not provided
I20260928 21:57:02.332561 18096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:57:02.332662 18096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:57:02.333739 18096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/master-0/data/instance:
uuid: "815c0c5647c04c23a589cb6f6d7e0aff"
format_stamp: "Formatted at 2026-09-28 21:57:02 on dist-test-slave-k6lz"
server_key: "8e03ffd46ff6ac356ad35c6e58bfad73"
server_key_iv: "4be693ad2925a889ba1d8538515d2523"
server_key_version: "encryptionkey@0"
I20260928 21:57:02.334064 18096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/master-0/wal/instance:
uuid: "815c0c5647c04c23a589cb6f6d7e0aff"
format_stamp: "Formatted at 2026-09-28 21:57:02 on dist-test-slave-k6lz"
server_key: "8e03ffd46ff6ac356ad35c6e58bfad73"
server_key_iv: "4be693ad2925a889ba1d8538515d2523"
server_key_version: "encryptionkey@0"
I20260928 21:57:02.335417 18096 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:57:02.336272 18111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:57:02.336462 18096 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:57:02.336552 18096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/master-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/master-0/wal
uuid: "815c0c5647c04c23a589cb6f6d7e0aff"
format_stamp: "Formatted at 2026-09-28 21:57:02 on dist-test-slave-k6lz"
server_key: "8e03ffd46ff6ac356ad35c6e58bfad73"
server_key_iv: "4be693ad2925a889ba1d8538515d2523"
server_key_version: "encryptionkey@0"
I20260928 21:57:02.336628 18096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:57:02.346814 18096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:57:02.348716 18096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:57:02.348834 18096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:57:02.352605 18096 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.126:42609
I20260928 21:57:02.352644 18163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.126:42609 every 8 connection(s)
I20260928 21:57:02.352964 18096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
I20260928 21:57:02.353823 18164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:57:02.356104 18164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 815c0c5647c04c23a589cb6f6d7e0aff: Bootstrap starting.
I20260928 21:57:02.356635 18164 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 815c0c5647c04c23a589cb6f6d7e0aff: Neither blocks nor log segments found. Creating new log.
I20260928 21:57:02.356865 18164 log.cc:824] T 00000000000000000000000000000000 P 815c0c5647c04c23a589cb6f6d7e0aff: Log is configured to *not* fsync() on all Append() calls
I20260928 21:57:02.357504 18164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 815c0c5647c04c23a589cb6f6d7e0aff: No bootstrap required, opened a new log
I20260928 21:57:02.359033 18164 raft_consensus.cc:348] T 00000000000000000000000000000000 P 815c0c5647c04c23a589cb6f6d7e0aff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "815c0c5647c04c23a589cb6f6d7e0aff" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 42609 } }
I20260928 21:57:02.359169 18164 raft_consensus.cc:374] T 00000000000000000000000000000000 P 815c0c5647c04c23a589cb6f6d7e0aff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:57:02.359190 18164 raft_consensus.cc:729] T 00000000000000000000000000000000 P 815c0c5647c04c23a589cb6f6d7e0aff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 815c0c5647c04c23a589cb6f6d7e0aff, State: Initialized, Role: FOLLOWER
I20260928 21:57:02.359294 18164 consensus_queue.cc:260] T 00000000000000000000000000000000 P 815c0c5647c04c23a589cb6f6d7e0aff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "815c0c5647c04c23a589cb6f6d7e0aff" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 42609 } }
I20260928 21:57:02.359386 18164 raft_consensus.cc:388] T 00000000000000000000000000000000 P 815c0c5647c04c23a589cb6f6d7e0aff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:57:02.359424 18164 raft_consensus.cc:482] T 00000000000000000000000000000000 P 815c0c5647c04c23a589cb6f6d7e0aff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:57:02.359470 18164 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 815c0c5647c04c23a589cb6f6d7e0aff [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:57:02.359973 18164 raft_consensus.cc:504] T 00000000000000000000000000000000 P 815c0c5647c04c23a589cb6f6d7e0aff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "815c0c5647c04c23a589cb6f6d7e0aff" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 42609 } }
I20260928 21:57:02.360061 18164 leader_election.cc:302] T 00000000000000000000000000000000 P 815c0c5647c04c23a589cb6f6d7e0aff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 815c0c5647c04c23a589cb6f6d7e0aff; no voters:
I20260928 21:57:02.360214 18164 leader_election.cc:288] T 00000000000000000000000000000000 P 815c0c5647c04c23a589cb6f6d7e0aff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:57:02.360270 18167 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 815c0c5647c04c23a589cb6f6d7e0aff [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:57:02.360422 18167 raft_consensus.cc:686] T 00000000000000000000000000000000 P 815c0c5647c04c23a589cb6f6d7e0aff [term 1 LEADER]: Becoming Leader. State: Replica: 815c0c5647c04c23a589cb6f6d7e0aff, State: Running, Role: LEADER
I20260928 21:57:02.360455 18164 sys_catalog.cc:564] T 00000000000000000000000000000000 P 815c0c5647c04c23a589cb6f6d7e0aff [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:57:02.360550 18167 consensus_queue.cc:237] T 00000000000000000000000000000000 P 815c0c5647c04c23a589cb6f6d7e0aff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "815c0c5647c04c23a589cb6f6d7e0aff" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 42609 } }
I20260928 21:57:02.360971 18169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 815c0c5647c04c23a589cb6f6d7e0aff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 815c0c5647c04c23a589cb6f6d7e0aff. Latest consensus state: current_term: 1 leader_uuid: "815c0c5647c04c23a589cb6f6d7e0aff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "815c0c5647c04c23a589cb6f6d7e0aff" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 42609 } } }
I20260928 21:57:02.360956 18168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 815c0c5647c04c23a589cb6f6d7e0aff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "815c0c5647c04c23a589cb6f6d7e0aff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "815c0c5647c04c23a589cb6f6d7e0aff" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 42609 } } }
I20260928 21:57:02.361145 18168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 815c0c5647c04c23a589cb6f6d7e0aff [sys.catalog]: This master's current role is: LEADER
I20260928 21:57:02.361176 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 18096
I20260928 21:57:02.361269 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/master-0/wal/instance
I20260928 21:57:02.361079 18169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 815c0c5647c04c23a589cb6f6d7e0aff [sys.catalog]: This master's current role is: LEADER
I20260928 21:57:02.361483 2777 external_mini_cluster.cc:1428] Setting key a429d5fe45dc861f40f9764472958759
I20260928 21:57:02.361564 18171 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:57:02.362058 18171 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:57:02.364239 18171 catalog_manager.cc:1347] Generated new cluster ID: 5e960ea181ec43699835cf6694426ea5
I20260928 21:57:02.364292 18171 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 21:57:02 dist-test-slave-k6lz krb5kdc[18083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:57:02.372069 18180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:02.362082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39314 (local address 127.2.182.126:42609)
0928 21:57:02.362445 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0928 21:57:02.362454 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 21:57:02.362491 (+ 37us) server_negotiation.cc:407] Connection header received
0928 21:57:02.362772 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:57:02.362781 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:57:02.362847 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:57:02.362987 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:57:02.363898 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:57:02.364963 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:57:02.365722 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:57:02.366228 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:57:02.366346 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:57:02.368748 (+ 2402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:57:02.368761 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:57:02.368761 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:57:02.368773 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:57:02.370294 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:57:02.370613 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:57:02.370616 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:57:02.370617 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:57:02.370702 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:57:02.371079 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:57:02.371080 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:57:02.371080 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:57:02.371442 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:57:02.371586 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 21:57:02.371688 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 21:57:02.371833 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":57,"threads_started":1}
I20260928 21:57:02.382215 18171 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:57:02.382762 18171 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:57:02.388406 18171 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 815c0c5647c04c23a589cb6f6d7e0aff: Generated new TSK 0
I20260928 21:57:02.388594 18171 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.182.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab.
Entry for principal kudu/127.2.182.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab.
I20260928 21:57:02.423439 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.65:0
--local_ip_for_outbound_sockets=127.2.182.65
--webserver_interface=127.2.182.65
--webserver_port=0
--tserver_master_addrs=127.2.182.126:42609
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab
--principal=kudu/127.2.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:41569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:57:02.518411 18190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:57:02.518637 18190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:57:02.518666 18190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:57:02.520020 18190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:57:02.520072 18190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:57:02.520100 18190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.65
I20260928 21:57:02.521608 18190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:41569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab
--principal=kudu/127.2.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.182.65
--webserver_port=0
--tserver_master_addrs=127.2.182.126:42609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18190
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.65
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:57:02.521834 18190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:57:02.522032 18190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:57:02.527585 18195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:57:02.527673 18196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:57:02.527678 18190 server_base.cc:1034] running on GCE node
W20260928 21:57:02.527585 18198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:57:02.528038 18190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:57:02.528263 18190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:57:02.529430 18190 hybrid_clock.cc:648] HybridClock initialized: now 1790632622529437 us; error 55 us; skew 500 ppm
Sep 28 21:57:02 dist-test-slave-k6lz krb5kdc[18083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632622, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:57:02.537287 18190 init.cc:377] Logged in from keytab as kudu/127.2.182.65@KRBTEST.COM (short username kudu)
I20260928 21:57:02.537799 18190 webserver.cc:460] Webserver started at http://127.2.182.65:41235/ using document root <none> and password file <none>
I20260928 21:57:02.538043 18190 fs_manager.cc:359] Metadata directory not provided
I20260928 21:57:02.538091 18190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:57:02.538201 18190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:57:02.539595 18190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-0/data/instance:
uuid: "508b43cb26334415a49e0c314db50f3d"
format_stamp: "Formatted at 2026-09-28 21:57:02 on dist-test-slave-k6lz"
server_key: "66e6dfbcdd213691d3dd0eb71920fdf7"
server_key_iv: "95af096eba8faf12a81ed76a744a1830"
server_key_version: "encryptionkey@0"
I20260928 21:57:02.539944 18190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-0/wal/instance:
uuid: "508b43cb26334415a49e0c314db50f3d"
format_stamp: "Formatted at 2026-09-28 21:57:02 on dist-test-slave-k6lz"
server_key: "66e6dfbcdd213691d3dd0eb71920fdf7"
server_key_iv: "95af096eba8faf12a81ed76a744a1830"
server_key_version: "encryptionkey@0"
I20260928 21:57:02.541388 18190 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 21:57:02.542366 18205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:57:02.542649 18190 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:57:02.542739 18190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-0/wal
uuid: "508b43cb26334415a49e0c314db50f3d"
format_stamp: "Formatted at 2026-09-28 21:57:02 on dist-test-slave-k6lz"
server_key: "66e6dfbcdd213691d3dd0eb71920fdf7"
server_key_iv: "95af096eba8faf12a81ed76a744a1830"
server_key_version: "encryptionkey@0"
I20260928 21:57:02.542819 18190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:57:02.551560 18190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:57:02.553757 18190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:57:02.553881 18190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:57:02.554248 18190 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:57:02.554569 18190 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:57:02.554605 18190 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:57:02.554634 18190 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:57:02.554659 18190 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:57:02.560534 18190 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.65:38691
I20260928 21:57:02.560581 18318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.65:38691 every 8 connection(s)
I20260928 21:57:02.560883 18190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
Sep 28 21:57:02 dist-test-slave-k6lz krb5kdc[18083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632622, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.65@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:57:02.569257 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 18190
I20260928 21:57:02.569373 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-0/wal/instance
I20260928 21:57:02.569562 2777 external_mini_cluster.cc:1428] Setting key 4cccf596f70b1cbbf9f7249d330ad7dd
I20260928 21:57:02.571141 18180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:02.561685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.65:58085 (local address 127.2.182.126:42609)
0928 21:57:02.561767 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 21:57:02.561769 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:57:02.561925 (+ 156us) server_negotiation.cc:407] Connection header received
0928 21:57:02.562628 (+ 703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:57:02.562629 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:57:02.562642 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:57:02.562684 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:57:02.563691 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:57:02.564339 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:57:02.565368 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:57:02.565573 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:57:02.565631 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:57:02.567860 (+ 2229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:57:02.567862 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:57:02.567862 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:57:02.567871 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:57:02.569344 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:57:02.569815 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:57:02.569817 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:57:02.569817 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:57:02.569884 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:57:02.570359 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:57:02.570360 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:57:02.570361 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:57:02.570516 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:57:02.570580 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 21:57:02.570993 (+ 413us) server_negotiation.cc:299] Negotiation successful
0928 21:57:02.571073 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 21:57:02.571403 18321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:02.561641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:42609 (local address 127.2.182.65:58085)
0928 21:57:02.561805 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:02.561842 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 21:57:02.562462 (+ 620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:57:02.562761 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:57:02.562764 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:57:02.563382 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:57:02.563591 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:57:02.563596 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:57:02.564453 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:57:02.564454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:57:02.565267 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:57:02.565269 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:57:02.565647 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:57:02.565647 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:57:02.565772 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:57:02.566328 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:57:02.566337 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:57:02.567746 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:57:02.569467 (+ 1721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:57:02.569476 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:57:02.569479 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:57:02.569743 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:57:02.570001 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:57:02.570002 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:57:02.570002 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:57:02.570143 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:57:02.570605 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:57:02.570613 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:57:02.570822 (+ 209us) client_negotiation.cc:769] Sending connection context
0928 21:57:02.570958 (+ 136us) client_negotiation.cc:240] Negotiation successful
0928 21:57:02.571132 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":77,"threads_started":1}
I20260928 21:57:02.572088 18319 heartbeater.cc:347] Connected to a master server at 127.2.182.126:42609
I20260928 21:57:02.572177 18319 heartbeater.cc:464] Registering TS with master...
I20260928 21:57:02.572368 18319 heartbeater.cc:511] Master 127.2.182.126:42609 requested a full tablet report, sending...
I20260928 21:57:02.572865 18128 ts_manager.cc:194] Registered new tserver with Master: 508b43cb26334415a49e0c314db50f3d (127.2.182.65:38691)
I20260928 21:57:02.573467 18128 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.65@KRBTEST.COM'} at 127.2.182.65:58085
WARNING: no policy specified for kudu/127.2.182.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab.
Entry for principal kudu/127.2.182.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab.
I20260928 21:57:02.595860 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.66:0
--local_ip_for_outbound_sockets=127.2.182.66
--webserver_interface=127.2.182.66
--webserver_port=0
--tserver_master_addrs=127.2.182.126:42609
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab
--principal=kudu/127.2.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:41569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:57:02.676986 18324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:57:02.677203 18324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:57:02.677227 18324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:57:02.678373 18324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:57:02.678413 18324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:57:02.678438 18324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.66
I20260928 21:57:02.679924 18324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:41569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab
--principal=kudu/127.2.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.182.66
--webserver_port=0
--tserver_master_addrs=127.2.182.126:42609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.66
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:57:02.680120 18324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:57:02.680315 18324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:57:02.685141 18332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:57:02.685214 18329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:57:02.685382 18330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:57:02.685401 18324 server_base.cc:1034] running on GCE node
I20260928 21:57:02.685784 18324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:57:02.686074 18324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:57:02.687311 18324 hybrid_clock.cc:648] HybridClock initialized: now 1790632622687293 us; error 28 us; skew 500 ppm
Sep 28 21:57:02 dist-test-slave-k6lz krb5kdc[18083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632622, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:57:02.689903 18324 init.cc:377] Logged in from keytab as kudu/127.2.182.66@KRBTEST.COM (short username kudu)
I20260928 21:57:02.690382 18324 webserver.cc:460] Webserver started at http://127.2.182.66:32807/ using document root <none> and password file <none>
I20260928 21:57:02.690578 18324 fs_manager.cc:359] Metadata directory not provided
I20260928 21:57:02.690620 18324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:57:02.690726 18324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:57:02.691880 18324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-1/data/instance:
uuid: "d1a41ac432214e4a8b4762b5cf41cce2"
format_stamp: "Formatted at 2026-09-28 21:57:02 on dist-test-slave-k6lz"
server_key: "f23ea8bf03b8d43d9756a4f8f7192561"
server_key_iv: "c509740ca6faa29346854111aaeb97c9"
server_key_version: "encryptionkey@0"
I20260928 21:57:02.692203 18324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-1/wal/instance:
uuid: "d1a41ac432214e4a8b4762b5cf41cce2"
format_stamp: "Formatted at 2026-09-28 21:57:02 on dist-test-slave-k6lz"
server_key: "f23ea8bf03b8d43d9756a4f8f7192561"
server_key_iv: "c509740ca6faa29346854111aaeb97c9"
server_key_version: "encryptionkey@0"
I20260928 21:57:02.693521 18324 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:57:02.694298 18339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:57:02.694484 18324 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:57:02.694566 18324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-1/wal
uuid: "d1a41ac432214e4a8b4762b5cf41cce2"
format_stamp: "Formatted at 2026-09-28 21:57:02 on dist-test-slave-k6lz"
server_key: "f23ea8bf03b8d43d9756a4f8f7192561"
server_key_iv: "c509740ca6faa29346854111aaeb97c9"
server_key_version: "encryptionkey@0"
I20260928 21:57:02.694626 18324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:57:02.705547 18324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:57:02.707800 18324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:57:02.707911 18324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:57:02.708096 18324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:57:02.708393 18324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:57:02.708427 18324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:57:02.708451 18324 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:57:02.708473 18324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:57:02.713985 18324 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.66:35109
I20260928 21:57:02.714022 18452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.66:35109 every 8 connection(s)
I20260928 21:57:02.714341 18324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
Sep 28 21:57:02 dist-test-slave-k6lz krb5kdc[18083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632622, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.66@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:57:02.720840 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 18324
I20260928 21:57:02.720980 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-1/wal/instance
I20260928 21:57:02.721202 2777 external_mini_cluster.cc:1428] Setting key d81482952992fe17bd7c8ed2dd330f4b
I20260928 21:57:02.724189 18180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:02.715039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.66:60233 (local address 127.2.182.126:42609)
0928 21:57:02.715122 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0928 21:57:02.715124 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:57:02.715286 (+ 162us) server_negotiation.cc:407] Connection header received
0928 21:57:02.715880 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:57:02.715882 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:57:02.715894 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:57:02.715943 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:57:02.716915 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:57:02.717417 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:57:02.718397 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:57:02.718611 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:57:02.718669 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:57:02.721131 (+ 2462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:57:02.721133 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:57:02.721133 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:57:02.721142 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:57:02.722663 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:57:02.723091 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:57:02.723092 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:57:02.723093 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:57:02.723155 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:57:02.723509 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:57:02.723510 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:57:02.723510 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:57:02.723656 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:57:02.723720 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 21:57:02.723987 (+ 267us) server_negotiation.cc:299] Negotiation successful
0928 21:57:02.724091 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 21:57:02.724304 18455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:02.714966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:42609 (local address 127.2.182.66:60233)
0928 21:57:02.715157 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:02.715198 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 21:57:02.715770 (+ 572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:57:02.716020 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:57:02.716023 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:57:02.716597 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:57:02.716792 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:57:02.716797 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:57:02.717511 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:57:02.717512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:57:02.718312 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:57:02.718315 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:57:02.718686 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:57:02.718687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:57:02.718807 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:57:02.719448 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:57:02.719457 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:57:02.721026 (+ 1569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:57:02.722775 (+ 1749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:57:02.722794 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:57:02.722795 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:57:02.723022 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:57:02.723212 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:57:02.723213 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:57:02.723213 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:57:02.723435 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:57:02.723710 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:57:02.723715 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:57:02.723856 (+ 141us) client_negotiation.cc:769] Sending connection context
0928 21:57:02.723942 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 21:57:02.724099 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":96,"threads_started":1}
I20260928 21:57:02.724763 18453 heartbeater.cc:347] Connected to a master server at 127.2.182.126:42609
I20260928 21:57:02.724862 18453 heartbeater.cc:464] Registering TS with master...
I20260928 21:57:02.725035 18453 heartbeater.cc:511] Master 127.2.182.126:42609 requested a full tablet report, sending...
I20260928 21:57:02.725395 18128 ts_manager.cc:194] Registered new tserver with Master: d1a41ac432214e4a8b4762b5cf41cce2 (127.2.182.66:35109)
I20260928 21:57:02.725911 18128 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.66@KRBTEST.COM'} at 127.2.182.66:60233
WARNING: no policy specified for kudu/127.2.182.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab.
Entry for principal kudu/127.2.182.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab.
I20260928 21:57:02.746454 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.67:0
--local_ip_for_outbound_sockets=127.2.182.67
--webserver_interface=127.2.182.67
--webserver_port=0
--tserver_master_addrs=127.2.182.126:42609
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab
--principal=kudu/127.2.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:41569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:57:02.826174 18458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:57:02.826392 18458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:57:02.826423 18458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:57:02.827592 18458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:57:02.827632 18458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:57:02.827666 18458 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.67
I20260928 21:57:02.829098 18458 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:41569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab
--principal=kudu/127.2.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.182.67
--webserver_port=0
--tserver_master_addrs=127.2.182.126:42609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.67
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:57:02.829337 18458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:57:02.829545 18458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:57:02.834685 18466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:57:02.834717 18463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:57:02.834685 18464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:57:02.834872 18458 server_base.cc:1034] running on GCE node
I20260928 21:57:02.835140 18458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:57:02.835505 18458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:57:02.836654 18458 hybrid_clock.cc:648] HybridClock initialized: now 1790632622836639 us; error 34 us; skew 500 ppm
Sep 28 21:57:02 dist-test-slave-k6lz krb5kdc[18083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632622, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:57:02.839157 18458 init.cc:377] Logged in from keytab as kudu/127.2.182.67@KRBTEST.COM (short username kudu)
I20260928 21:57:02.839692 18458 webserver.cc:460] Webserver started at http://127.2.182.67:41839/ using document root <none> and password file <none>
I20260928 21:57:02.839910 18458 fs_manager.cc:359] Metadata directory not provided
I20260928 21:57:02.839954 18458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:57:02.840083 18458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:57:02.841125 18458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-2/data/instance:
uuid: "2bcae726249442b885908d0a151ac987"
format_stamp: "Formatted at 2026-09-28 21:57:02 on dist-test-slave-k6lz"
server_key: "8a7655b4c4a42bdfd9db00aca4487736"
server_key_iv: "de650a7d630e4ad987a8de22b78cdb48"
server_key_version: "encryptionkey@0"
I20260928 21:57:02.841470 18458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-2/wal/instance:
uuid: "2bcae726249442b885908d0a151ac987"
format_stamp: "Formatted at 2026-09-28 21:57:02 on dist-test-slave-k6lz"
server_key: "8a7655b4c4a42bdfd9db00aca4487736"
server_key_iv: "de650a7d630e4ad987a8de22b78cdb48"
server_key_version: "encryptionkey@0"
I20260928 21:57:02.842706 18458 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:57:02.843521 18473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:57:02.843672 18458 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:57:02.843787 18458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-2/wal
uuid: "2bcae726249442b885908d0a151ac987"
format_stamp: "Formatted at 2026-09-28 21:57:02 on dist-test-slave-k6lz"
server_key: "8a7655b4c4a42bdfd9db00aca4487736"
server_key_iv: "de650a7d630e4ad987a8de22b78cdb48"
server_key_version: "encryptionkey@0"
I20260928 21:57:02.843860 18458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:57:02.855813 18458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:57:02.857874 18458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:57:02.858016 18458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:57:02.858228 18458 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:57:02.858528 18458 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:57:02.858561 18458 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:57:02.858590 18458 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:57:02.858608 18458 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:57:02.864598 18458 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.67:46537
I20260928 21:57:02.864681 18586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.67:46537 every 8 connection(s)
I20260928 21:57:02.864966 18458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
Sep 28 21:57:02 dist-test-slave-k6lz krb5kdc[18083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632622, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.67@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:57:02.871435 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 18458
I20260928 21:57:02.871543 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-2/wal/instance
I20260928 21:57:02.871768 2777 external_mini_cluster.cc:1428] Setting key a05c7f9eee8e01f5f3f12a868e625d1c
I20260928 21:57:02.874641 18180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:02.865704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.67:58881 (local address 127.2.182.126:42609)
0928 21:57:02.865750 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 21:57:02.865751 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:57:02.865952 (+ 201us) server_negotiation.cc:407] Connection header received
0928 21:57:02.866884 (+ 932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:57:02.866886 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:57:02.866897 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:57:02.866963 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:57:02.867608 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:57:02.868136 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:57:02.869101 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:57:02.869291 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:57:02.869349 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:57:02.871738 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:57:02.871740 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:57:02.871740 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:57:02.871754 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:57:02.873210 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:57:02.873648 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:57:02.873650 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:57:02.873650 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:57:02.873713 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:57:02.874025 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:57:02.874026 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:57:02.874026 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:57:02.874160 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:57:02.874215 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 21:57:02.874489 (+ 274us) server_negotiation.cc:299] Negotiation successful
0928 21:57:02.874565 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 21:57:02.874752 18589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:02.865657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:42609 (local address 127.2.182.67:58881)
0928 21:57:02.865799 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:02.865841 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 21:57:02.866775 (+ 934us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:57:02.867019 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:57:02.867022 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:57:02.867263 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:57:02.867525 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:57:02.867530 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:57:02.868214 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:57:02.868215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:57:02.869011 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:57:02.869014 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:57:02.869362 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:57:02.869363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:57:02.869463 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:57:02.869937 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:57:02.869945 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:57:02.871630 (+ 1685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:57:02.873321 (+ 1691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:57:02.873328 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:57:02.873330 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:57:02.873567 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:57:02.873802 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:57:02.873803 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:57:02.873803 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:57:02.873938 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:57:02.874208 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:57:02.874212 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:57:02.874333 (+ 121us) client_negotiation.cc:769] Sending connection context
0928 21:57:02.874433 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 21:57:02.874569 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":74,"threads_started":1}
I20260928 21:57:02.875183 18587 heartbeater.cc:347] Connected to a master server at 127.2.182.126:42609
I20260928 21:57:02.875272 18587 heartbeater.cc:464] Registering TS with master...
I20260928 21:57:02.875470 18587 heartbeater.cc:511] Master 127.2.182.126:42609 requested a full tablet report, sending...
I20260928 21:57:02.875852 18128 ts_manager.cc:194] Registered new tserver with Master: 2bcae726249442b885908d0a151ac987 (127.2.182.67:46537)
I20260928 21:57:02.876188 18128 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.67@KRBTEST.COM'} at 127.2.182.67:58881
I20260928 21:57:02.883740 2777 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:57:02.891021 18180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:02.885063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39330 (local address 127.2.182.126:42609)
0928 21:57:02.885117 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 21:57:02.885119 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:57:02.885198 (+ 79us) server_negotiation.cc:407] Connection header received
0928 21:57:02.885250 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:57:02.885251 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:57:02.885260 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:57:02.885307 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:57:02.885789 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:57:02.886270 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:57:02.887042 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:57:02.887232 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:57:02.887289 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:57:02.888574 (+ 1285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:57:02.888576 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:57:02.888577 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:57:02.888596 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:57:02.889940 (+ 1344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:57:02.890348 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:57:02.890349 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:57:02.890350 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:57:02.890409 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:57:02.890576 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:57:02.890576 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:57:02.890576 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:57:02.890722 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:57:02.890793 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 21:57:02.890893 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 21:57:02.890954 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 21:57:02.892652 18128 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39330:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:57:02.894245 18128 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:57:02.900873 18603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:02.897067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34264 (local address 127.2.182.66:35109)
0928 21:57:02.897334 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0928 21:57:02.897339 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:57:02.897347 (+ 8us) server_negotiation.cc:407] Connection header received
0928 21:57:02.897383 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:57:02.897385 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:57:02.897400 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:57:02.897523 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:57:02.897628 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:57:02.898694 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:57:02.899814 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:57:02.900471 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:57:02.900547 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:57:02.900606 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 21:57:02.900741 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 21:57:02.900784 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":75,"threads_started":1}
I20260928 21:57:02.900867 18599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:02.896516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.66:35109 (local address 127.0.0.1:34264)
0928 21:57:02.896736 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:02.896746 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:57:02.896800 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:57:02.897525 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:57:02.897526 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:57:02.897528 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:57:02.897575 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:57:02.897576 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:57:02.898779 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:57:02.898780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:57:02.899737 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:57:02.899740 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:57:02.900559 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:57:02.900560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:57:02.900636 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:57:02.900650 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 21:57:02.900716 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 21:57:02.900780 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":94,"threads_started":1}
I20260928 21:57:02.901028 18601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:02.896836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56378 (local address 127.2.182.65:38691)
0928 21:57:02.897028 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 21:57:02.897032 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:57:02.897437 (+ 405us) server_negotiation.cc:407] Connection header received
0928 21:57:02.897480 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:57:02.897482 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:57:02.897498 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:57:02.897559 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:57:02.897733 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:57:02.898513 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:57:02.900065 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:57:02.900564 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:57:02.900629 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:57:02.900683 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 21:57:02.900909 (+ 226us) server_negotiation.cc:299] Negotiation successful
0928 21:57:02.900952 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":83,"threads_started":1}
I20260928 21:57:02.901046 18598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:02.896490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.67:46537 (local address 127.0.0.1:37786)
0928 21:57:02.896623 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:02.896665 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 21:57:02.896724 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:57:02.897220 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:57:02.897223 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:57:02.897225 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:57:02.897304 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:57:02.897306 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:57:02.898662 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:57:02.898662 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:57:02.899971 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:57:02.899973 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:57:02.900821 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:57:02.900822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:57:02.900894 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:57:02.900900 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 21:57:02.900928 (+ 28us) client_negotiation.cc:240] Negotiation successful
0928 21:57:02.900971 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":61,"thread_start_us":51,"threads_started":1}
I20260928 21:57:02.901149 18600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:02.896705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.65:38691 (local address 127.0.0.1:56378)
0928 21:57:02.897228 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:02.897238 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:57:02.897376 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:57:02.897628 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:57:02.897629 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:57:02.897630 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:57:02.897669 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:57:02.897670 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:57:02.898601 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:57:02.898602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:57:02.899970 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:57:02.899973 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:57:02.900714 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:57:02.900715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:57:02.900769 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:57:02.900772 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:57:02.900824 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 21:57:02.900860 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"mutex_wait_us":45,"thread_start_us":72,"threads_started":1}
I20260928 21:57:02.901432 18602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:02.896854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37786 (local address 127.2.182.67:46537)
0928 21:57:02.897035 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 21:57:02.897039 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:57:02.897048 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:57:02.897081 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:57:02.897082 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:57:02.897095 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:57:02.897383 (+ 288us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:57:02.897455 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:57:02.898572 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:57:02.900050 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:57:02.900730 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:57:02.901222 (+ 492us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:57:02.901288 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 21:57:02.901313 (+ 25us) server_negotiation.cc:299] Negotiation successful
0928 21:57:02.901355 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":93,"threads_started":1}
I20260928 21:57:02.901878 18253 tablet_service.cc:1467] Processing CreateTablet for tablet 535438832fe04650857cbf03633e6864 (DEFAULT_TABLE table=test-table [id=dc59cdfb3a15438db2be21f210d3ab9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:57:02.902210 18253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 535438832fe04650857cbf03633e6864. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:57:02.904011 18387 tablet_service.cc:1467] Processing CreateTablet for tablet 535438832fe04650857cbf03633e6864 (DEFAULT_TABLE table=test-table [id=dc59cdfb3a15438db2be21f210d3ab9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:57:02.904072 18521 tablet_service.cc:1467] Processing CreateTablet for tablet 535438832fe04650857cbf03633e6864 (DEFAULT_TABLE table=test-table [id=dc59cdfb3a15438db2be21f210d3ab9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:57:02.904306 18387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 535438832fe04650857cbf03633e6864. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:57:02.904388 18521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 535438832fe04650857cbf03633e6864. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:57:02.905205 18604 tablet_bootstrap.cc:492] T 535438832fe04650857cbf03633e6864 P 508b43cb26334415a49e0c314db50f3d: Bootstrap starting.
I20260928 21:57:02.905871 18604 tablet_bootstrap.cc:654] T 535438832fe04650857cbf03633e6864 P 508b43cb26334415a49e0c314db50f3d: Neither blocks nor log segments found. Creating new log.
I20260928 21:57:02.906494 18604 log.cc:824] T 535438832fe04650857cbf03633e6864 P 508b43cb26334415a49e0c314db50f3d: Log is configured to *not* fsync() on all Append() calls
I20260928 21:57:02.907159 18606 tablet_bootstrap.cc:492] T 535438832fe04650857cbf03633e6864 P 2bcae726249442b885908d0a151ac987: Bootstrap starting.
I20260928 21:57:02.907228 18604 tablet_bootstrap.cc:492] T 535438832fe04650857cbf03633e6864 P 508b43cb26334415a49e0c314db50f3d: No bootstrap required, opened a new log
I20260928 21:57:02.907290 18604 ts_tablet_manager.cc:1397] T 535438832fe04650857cbf03633e6864 P 508b43cb26334415a49e0c314db50f3d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 21:57:02.907500 18607 tablet_bootstrap.cc:492] T 535438832fe04650857cbf03633e6864 P d1a41ac432214e4a8b4762b5cf41cce2: Bootstrap starting.
I20260928 21:57:02.907792 18606 tablet_bootstrap.cc:654] T 535438832fe04650857cbf03633e6864 P 2bcae726249442b885908d0a151ac987: Neither blocks nor log segments found. Creating new log.
I20260928 21:57:02.908087 18607 tablet_bootstrap.cc:654] T 535438832fe04650857cbf03633e6864 P d1a41ac432214e4a8b4762b5cf41cce2: Neither blocks nor log segments found. Creating new log.
I20260928 21:57:02.908305 18606 log.cc:824] T 535438832fe04650857cbf03633e6864 P 2bcae726249442b885908d0a151ac987: Log is configured to *not* fsync() on all Append() calls
I20260928 21:57:02.908630 18607 log.cc:824] T 535438832fe04650857cbf03633e6864 P d1a41ac432214e4a8b4762b5cf41cce2: Log is configured to *not* fsync() on all Append() calls
I20260928 21:57:02.908987 18606 tablet_bootstrap.cc:492] T 535438832fe04650857cbf03633e6864 P 2bcae726249442b885908d0a151ac987: No bootstrap required, opened a new log
I20260928 21:57:02.909063 18606 ts_tablet_manager.cc:1397] T 535438832fe04650857cbf03633e6864 P 2bcae726249442b885908d0a151ac987: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:57:02.909409 18607 tablet_bootstrap.cc:492] T 535438832fe04650857cbf03633e6864 P d1a41ac432214e4a8b4762b5cf41cce2: No bootstrap required, opened a new log
I20260928 21:57:02.909283 18604 raft_consensus.cc:348] T 535438832fe04650857cbf03633e6864 P 508b43cb26334415a49e0c314db50f3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bcae726249442b885908d0a151ac987" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 46537 } } peers { permanent_uuid: "d1a41ac432214e4a8b4762b5cf41cce2" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 35109 } } peers { permanent_uuid: "508b43cb26334415a49e0c314db50f3d" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 38691 } }
I20260928 21:57:02.909479 18604 raft_consensus.cc:374] T 535438832fe04650857cbf03633e6864 P 508b43cb26334415a49e0c314db50f3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:57:02.909492 18607 ts_tablet_manager.cc:1397] T 535438832fe04650857cbf03633e6864 P d1a41ac432214e4a8b4762b5cf41cce2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:57:02.909525 18604 raft_consensus.cc:729] T 535438832fe04650857cbf03633e6864 P 508b43cb26334415a49e0c314db50f3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 508b43cb26334415a49e0c314db50f3d, State: Initialized, Role: FOLLOWER
I20260928 21:57:02.909657 18604 consensus_queue.cc:260] T 535438832fe04650857cbf03633e6864 P 508b43cb26334415a49e0c314db50f3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bcae726249442b885908d0a151ac987" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 46537 } } peers { permanent_uuid: "d1a41ac432214e4a8b4762b5cf41cce2" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 35109 } } peers { permanent_uuid: "508b43cb26334415a49e0c314db50f3d" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 38691 } }
I20260928 21:57:02.910022 18604 ts_tablet_manager.cc:1428] T 535438832fe04650857cbf03633e6864 P 508b43cb26334415a49e0c314db50f3d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 21:57:02.910171 18319 heartbeater.cc:503] Master 127.2.182.126:42609 was elected leader, sending a full tablet report...
I20260928 21:57:02.911104 18606 raft_consensus.cc:348] T 535438832fe04650857cbf03633e6864 P 2bcae726249442b885908d0a151ac987 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bcae726249442b885908d0a151ac987" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 46537 } } peers { permanent_uuid: "d1a41ac432214e4a8b4762b5cf41cce2" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 35109 } } peers { permanent_uuid: "508b43cb26334415a49e0c314db50f3d" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 38691 } }
I20260928 21:57:02.911247 18606 raft_consensus.cc:374] T 535438832fe04650857cbf03633e6864 P 2bcae726249442b885908d0a151ac987 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:57:02.911278 18606 raft_consensus.cc:729] T 535438832fe04650857cbf03633e6864 P 2bcae726249442b885908d0a151ac987 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bcae726249442b885908d0a151ac987, State: Initialized, Role: FOLLOWER
I20260928 21:57:02.911429 18606 consensus_queue.cc:260] T 535438832fe04650857cbf03633e6864 P 2bcae726249442b885908d0a151ac987 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bcae726249442b885908d0a151ac987" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 46537 } } peers { permanent_uuid: "d1a41ac432214e4a8b4762b5cf41cce2" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 35109 } } peers { permanent_uuid: "508b43cb26334415a49e0c314db50f3d" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 38691 } }
I20260928 21:57:02.911710 18587 heartbeater.cc:503] Master 127.2.182.126:42609 was elected leader, sending a full tablet report...
I20260928 21:57:02.911721 18606 ts_tablet_manager.cc:1428] T 535438832fe04650857cbf03633e6864 P 2bcae726249442b885908d0a151ac987: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 21:57:02.911617 18607 raft_consensus.cc:348] T 535438832fe04650857cbf03633e6864 P d1a41ac432214e4a8b4762b5cf41cce2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bcae726249442b885908d0a151ac987" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 46537 } } peers { permanent_uuid: "d1a41ac432214e4a8b4762b5cf41cce2" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 35109 } } peers { permanent_uuid: "508b43cb26334415a49e0c314db50f3d" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 38691 } }
I20260928 21:57:02.911839 18607 raft_consensus.cc:374] T 535438832fe04650857cbf03633e6864 P d1a41ac432214e4a8b4762b5cf41cce2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:57:02.911878 18607 raft_consensus.cc:729] T 535438832fe04650857cbf03633e6864 P d1a41ac432214e4a8b4762b5cf41cce2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a41ac432214e4a8b4762b5cf41cce2, State: Initialized, Role: FOLLOWER
I20260928 21:57:02.911998 18607 consensus_queue.cc:260] T 535438832fe04650857cbf03633e6864 P d1a41ac432214e4a8b4762b5cf41cce2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bcae726249442b885908d0a151ac987" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 46537 } } peers { permanent_uuid: "d1a41ac432214e4a8b4762b5cf41cce2" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 35109 } } peers { permanent_uuid: "508b43cb26334415a49e0c314db50f3d" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 38691 } }
I20260928 21:57:02.912277 18607 ts_tablet_manager.cc:1428] T 535438832fe04650857cbf03633e6864 P d1a41ac432214e4a8b4762b5cf41cce2: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 21:57:02.912302 18453 heartbeater.cc:503] Master 127.2.182.126:42609 was elected leader, sending a full tablet report...
I20260928 21:57:02.962373 18612 raft_consensus.cc:482] T 535438832fe04650857cbf03633e6864 P d1a41ac432214e4a8b4762b5cf41cce2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:57:02.962508 18612 raft_consensus.cc:504] T 535438832fe04650857cbf03633e6864 P d1a41ac432214e4a8b4762b5cf41cce2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bcae726249442b885908d0a151ac987" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 46537 } } peers { permanent_uuid: "d1a41ac432214e4a8b4762b5cf41cce2" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 35109 } } peers { permanent_uuid: "508b43cb26334415a49e0c314db50f3d" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 38691 } }
I20260928 21:57:02.962950 18612 leader_election.cc:288] T 535438832fe04650857cbf03633e6864 P d1a41ac432214e4a8b4762b5cf41cce2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2bcae726249442b885908d0a151ac987 (127.2.182.67:46537), 508b43cb26334415a49e0c314db50f3d (127.2.182.65:38691)
W20260928 21:57:02.964849 18454 tablet.cc:2369] T 535438832fe04650857cbf03633e6864 P d1a41ac432214e4a8b4762b5cf41cce2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:57:02.966979 18602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:02.963363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.66:37637 (local address 127.2.182.67:46537)
0928 21:57:02.963419 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 21:57:02.963421 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:57:02.963428 (+ 7us) server_negotiation.cc:407] Connection header received
0928 21:57:02.963452 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:57:02.963453 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:57:02.963461 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:57:02.963515 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:57:02.963694 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:57:02.964471 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:57:02.965950 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:57:02.966600 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:57:02.966662 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:57:02.966670 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 21:57:02.966865 (+ 195us) server_negotiation.cc:299] Negotiation successful
0928 21:57:02.966905 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 21:57:02.966979 18601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:02.963228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.66:51183 (local address 127.2.182.65:38691)
0928 21:57:02.963272 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 21:57:02.963274 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:57:02.963489 (+ 215us) server_negotiation.cc:407] Connection header received
0928 21:57:02.963523 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:57:02.963524 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:57:02.963533 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:57:02.963583 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:57:02.963708 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:57:02.964467 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:57:02.965950 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:57:02.966555 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:57:02.966641 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:57:02.966689 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 21:57:02.966865 (+ 176us) server_negotiation.cc:299] Negotiation successful
0928 21:57:02.966907 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 21:57:02.966995 18455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:02.963154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.67:46537 (local address 127.2.182.66:37637)
0928 21:57:02.963214 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:02.963226 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:57:02.963278 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:57:02.963540 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:57:02.963541 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:57:02.963543 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:57:02.963623 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:57:02.963626 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:57:02.964587 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:57:02.964588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:57:02.965836 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:57:02.965840 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:57:02.966698 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:57:02.966699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:57:02.966790 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:57:02.966792 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 21:57:02.966855 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 21:57:02.966901 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":33}
I20260928 21:57:02.967007 18613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:02.963201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.65:38691 (local address 127.2.182.66:51183)
0928 21:57:02.963404 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:02.963421 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 21:57:02.963479 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:57:02.963592 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:57:02.963593 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:57:02.963594 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:57:02.963651 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:57:02.963652 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:57:02.964587 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:57:02.964588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:57:02.965836 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:57:02.965840 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:57:02.966706 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:57:02.966707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:57:02.966784 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:57:02.966790 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 21:57:02.966852 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 21:57:02.966901 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":89,"thread_start_us":116,"threads_started":1}
I20260928 21:57:02.967478 18541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "535438832fe04650857cbf03633e6864" candidate_uuid: "d1a41ac432214e4a8b4762b5cf41cce2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bcae726249442b885908d0a151ac987" is_pre_election: true
I20260928 21:57:02.967545 18273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "535438832fe04650857cbf03633e6864" candidate_uuid: "d1a41ac432214e4a8b4762b5cf41cce2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "508b43cb26334415a49e0c314db50f3d" is_pre_election: true
I20260928 21:57:02.967613 18541 raft_consensus.cc:2454] T 535438832fe04650857cbf03633e6864 P 2bcae726249442b885908d0a151ac987 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1a41ac432214e4a8b4762b5cf41cce2 in term 0.
I20260928 21:57:02.967701 18273 raft_consensus.cc:2454] T 535438832fe04650857cbf03633e6864 P 508b43cb26334415a49e0c314db50f3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1a41ac432214e4a8b4762b5cf41cce2 in term 0.
I20260928 21:57:02.967869 18341 leader_election.cc:302] T 535438832fe04650857cbf03633e6864 P d1a41ac432214e4a8b4762b5cf41cce2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bcae726249442b885908d0a151ac987, d1a41ac432214e4a8b4762b5cf41cce2; no voters:
I20260928 21:57:02.967989 18612 raft_consensus.cc:2781] T 535438832fe04650857cbf03633e6864 P d1a41ac432214e4a8b4762b5cf41cce2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:57:02.968044 18612 raft_consensus.cc:482] T 535438832fe04650857cbf03633e6864 P d1a41ac432214e4a8b4762b5cf41cce2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:57:02.968066 18612 raft_consensus.cc:3037] T 535438832fe04650857cbf03633e6864 P d1a41ac432214e4a8b4762b5cf41cce2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:57:02.968734 18612 raft_consensus.cc:504] T 535438832fe04650857cbf03633e6864 P d1a41ac432214e4a8b4762b5cf41cce2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bcae726249442b885908d0a151ac987" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 46537 } } peers { permanent_uuid: "d1a41ac432214e4a8b4762b5cf41cce2" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 35109 } } peers { permanent_uuid: "508b43cb26334415a49e0c314db50f3d" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 38691 } }
I20260928 21:57:02.968917 18612 leader_election.cc:288] T 535438832fe04650857cbf03633e6864 P d1a41ac432214e4a8b4762b5cf41cce2 [CANDIDATE]: Term 1 election: Requested vote from peers 2bcae726249442b885908d0a151ac987 (127.2.182.67:46537), 508b43cb26334415a49e0c314db50f3d (127.2.182.65:38691)
I20260928 21:57:02.969002 18273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "535438832fe04650857cbf03633e6864" candidate_uuid: "d1a41ac432214e4a8b4762b5cf41cce2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "508b43cb26334415a49e0c314db50f3d"
I20260928 21:57:02.969058 18541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "535438832fe04650857cbf03633e6864" candidate_uuid: "d1a41ac432214e4a8b4762b5cf41cce2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bcae726249442b885908d0a151ac987"
I20260928 21:57:02.969089 18273 raft_consensus.cc:3037] T 535438832fe04650857cbf03633e6864 P 508b43cb26334415a49e0c314db50f3d [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:57:02.969126 18541 raft_consensus.cc:3037] T 535438832fe04650857cbf03633e6864 P 2bcae726249442b885908d0a151ac987 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:57:02.969758 18541 raft_consensus.cc:2454] T 535438832fe04650857cbf03633e6864 P 2bcae726249442b885908d0a151ac987 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1a41ac432214e4a8b4762b5cf41cce2 in term 1.
I20260928 21:57:02.969857 18273 raft_consensus.cc:2454] T 535438832fe04650857cbf03633e6864 P 508b43cb26334415a49e0c314db50f3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1a41ac432214e4a8b4762b5cf41cce2 in term 1.
I20260928 21:57:02.969946 18341 leader_election.cc:302] T 535438832fe04650857cbf03633e6864 P d1a41ac432214e4a8b4762b5cf41cce2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bcae726249442b885908d0a151ac987, d1a41ac432214e4a8b4762b5cf41cce2; no voters:
I20260928 21:57:02.970036 18612 raft_consensus.cc:2781] T 535438832fe04650857cbf03633e6864 P d1a41ac432214e4a8b4762b5cf41cce2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:57:02.970220 18612 raft_consensus.cc:686] T 535438832fe04650857cbf03633e6864 P d1a41ac432214e4a8b4762b5cf41cce2 [term 1 LEADER]: Becoming Leader. State: Replica: d1a41ac432214e4a8b4762b5cf41cce2, State: Running, Role: LEADER
I20260928 21:57:02.970311 18612 consensus_queue.cc:237] T 535438832fe04650857cbf03633e6864 P d1a41ac432214e4a8b4762b5cf41cce2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bcae726249442b885908d0a151ac987" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 46537 } } peers { permanent_uuid: "d1a41ac432214e4a8b4762b5cf41cce2" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 35109 } } peers { permanent_uuid: "508b43cb26334415a49e0c314db50f3d" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 38691 } }
I20260928 21:57:02.971287 18127 catalog_manager.cc:5495] T 535438832fe04650857cbf03633e6864 P d1a41ac432214e4a8b4762b5cf41cce2 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1a41ac432214e4a8b4762b5cf41cce2 (127.2.182.66). New cstate: current_term: 1 leader_uuid: "d1a41ac432214e4a8b4762b5cf41cce2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bcae726249442b885908d0a151ac987" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 46537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1a41ac432214e4a8b4762b5cf41cce2" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 35109 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "508b43cb26334415a49e0c314db50f3d" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 38691 } health_report { overall_health: UNKNOWN } } }
I20260928 21:57:02.995579 18603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:02.993030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34278 (local address 127.2.182.66:35109)
0928 21:57:02.993086 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 21:57:02.993087 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:57:02.993095 (+ 8us) server_negotiation.cc:407] Connection header received
0928 21:57:02.993134 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:57:02.993135 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:57:02.993151 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:57:02.993286 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:57:02.993407 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:57:02.993884 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:57:02.994754 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:57:02.994964 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:57:02.995025 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:57:02.995222 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:57:02.995355 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:57:02.995437 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 21:57:02.995476 (+ 39us) server_negotiation.cc:299] Negotiation successful
0928 21:57:02.995498 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 21:57:02.997466 18541 raft_consensus.cc:1261] T 535438832fe04650857cbf03633e6864 P 2bcae726249442b885908d0a151ac987 [term 1 FOLLOWER]: Refusing update from remote peer d1a41ac432214e4a8b4762b5cf41cce2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:57:02.997486 18273 raft_consensus.cc:1261] T 535438832fe04650857cbf03633e6864 P 508b43cb26334415a49e0c314db50f3d [term 1 FOLLOWER]: Refusing update from remote peer d1a41ac432214e4a8b4762b5cf41cce2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:57:02.997833 18614 consensus_queue.cc:1035] T 535438832fe04650857cbf03633e6864 P d1a41ac432214e4a8b4762b5cf41cce2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bcae726249442b885908d0a151ac987" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 46537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:57:02.997970 18612 consensus_queue.cc:1035] T 535438832fe04650857cbf03633e6864 P d1a41ac432214e4a8b4762b5cf41cce2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "508b43cb26334415a49e0c314db50f3d" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 38691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:57:03.000277 18617 mvcc.cc:205] Tried to move back new op lower bound from 7334431223794958336 to 7334431223686983680. Current Snapshot: MvccSnapshot[applied={T|T < 7334431223794958336}]
I20260928 21:57:03.002544 18128 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:57:03.002638 18128 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:57:03.003415 18128 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 815c0c5647c04c23a589cb6f6d7e0aff: Sending DeleteTablet for 3 replicas of tablet 535438832fe04650857cbf03633e6864
I20260928 21:57:03.003726 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 18190
I20260928 21:57:03.003789 18521 tablet_service.cc:1514] Processing DeleteTablet for tablet 535438832fe04650857cbf03633e6864 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:57:02 UTC) from {username='kudu', principal='kudu/127.2.182.126@KRBTEST.COM'} at 127.0.0.1:37786
I20260928 21:57:03.004001 18387 tablet_service.cc:1514] Processing DeleteTablet for tablet 535438832fe04650857cbf03633e6864 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:57:02 UTC) from {username='kudu', principal='kudu/127.2.182.126@KRBTEST.COM'} at 127.0.0.1:34264
I20260928 21:57:03.004407 18627 tablet_replica.cc:330] T 535438832fe04650857cbf03633e6864 P d1a41ac432214e4a8b4762b5cf41cce2: stopping tablet replica
I20260928 21:57:03.004505 18627 raft_consensus.cc:2229] T 535438832fe04650857cbf03633e6864 P d1a41ac432214e4a8b4762b5cf41cce2 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:57:03.004599 18627 raft_consensus.cc:2258] T 535438832fe04650857cbf03633e6864 P d1a41ac432214e4a8b4762b5cf41cce2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:57:03.005014 18627 ts_tablet_manager.cc:1905] T 535438832fe04650857cbf03633e6864 P d1a41ac432214e4a8b4762b5cf41cce2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:57:03.006913 18627 ts_tablet_manager.cc:1918] T 535438832fe04650857cbf03633e6864 P d1a41ac432214e4a8b4762b5cf41cce2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:57:03.006992 18627 log.cc:1196] T 535438832fe04650857cbf03633e6864 P d1a41ac432214e4a8b4762b5cf41cce2: Deleting WAL directory at /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-1/wal/wals/535438832fe04650857cbf03633e6864
I20260928 21:57:03.007223 18627 ts_tablet_manager.cc:1939] T 535438832fe04650857cbf03633e6864 P d1a41ac432214e4a8b4762b5cf41cce2: Deleting consensus metadata
I20260928 21:57:03.007690 18115 catalog_manager.cc:4841] TS d1a41ac432214e4a8b4762b5cf41cce2 (127.2.182.66:35109): tablet 535438832fe04650857cbf03633e6864 (table test-table [id=dc59cdfb3a15438db2be21f210d3ab9c]) successfully deleted
W20260928 21:57:03.009902 18115 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.182.65:38691 (error 108)
I20260928 21:57:03.010102 18626 tablet_replica.cc:330] T 535438832fe04650857cbf03633e6864 P 2bcae726249442b885908d0a151ac987: stopping tablet replica
I20260928 21:57:03.010233 18626 raft_consensus.cc:2229] T 535438832fe04650857cbf03633e6864 P 2bcae726249442b885908d0a151ac987 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:57:03.010372 18626 raft_consensus.cc:2258] T 535438832fe04650857cbf03633e6864 P 2bcae726249442b885908d0a151ac987 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:57:03.010464 18600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:03.010340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.65:38691 (local address 127.0.0.1:56390)
0928 21:57:03.010381 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:03.010425 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.65:38691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260928 21:57:03.010572 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 18324
W20260928 21:57:03.010579 18115 catalog_manager.cc:4584] TS 508b43cb26334415a49e0c314db50f3d (127.2.182.65:38691): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 535438832fe04650857cbf03633e6864: Network error: Client connection negotiation failed: client connection to 127.2.182.65:38691: connect: Connection refused (error 111)
I20260928 21:57:03.011004 18626 ts_tablet_manager.cc:1905] T 535438832fe04650857cbf03633e6864 P 2bcae726249442b885908d0a151ac987: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:57:03.013113 18626 ts_tablet_manager.cc:1918] T 535438832fe04650857cbf03633e6864 P 2bcae726249442b885908d0a151ac987: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:57:03.013262 18626 log.cc:1196] T 535438832fe04650857cbf03633e6864 P 2bcae726249442b885908d0a151ac987: Deleting WAL directory at /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-2/wal/wals/535438832fe04650857cbf03633e6864
I20260928 21:57:03.013495 18626 ts_tablet_manager.cc:1939] T 535438832fe04650857cbf03633e6864 P 2bcae726249442b885908d0a151ac987: Deleting consensus metadata
I20260928 21:57:03.013970 18113 catalog_manager.cc:4841] TS 2bcae726249442b885908d0a151ac987 (127.2.182.67:46537): tablet 535438832fe04650857cbf03633e6864 (table test-table [id=dc59cdfb3a15438db2be21f210d3ab9c]) successfully deleted
I20260928 21:57:03.015355 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 18458
I20260928 21:57:03.019423 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 18096
I20260928 21:57:23.024430 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:42609
--webserver_interface=127.2.182.126
--webserver_port=46749
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab
--principal=kudu/127.2.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:41569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.126:42609
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:57:23.106925 18629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:57:23.107159 18629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:57:23.107203 18629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:57:23.108486 18629 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 21:57:23.108537 18629 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 21:57:23.108554 18629 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 21:57:23.108569 18629 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:57:23.108587 18629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:57:23.108603 18629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:57:23.108619 18629 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:57:23.108640 18629 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:57:23.110263 18629 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:41569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.182.126:42609
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab
--principal=kudu/127.2.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:42609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.126
--webserver_port=46749
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:57:23.110544 18629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:57:23.110775 18629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:57:23.120456 18638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:57:23.120456 18635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:57:23.120605 18629 server_base.cc:1034] running on GCE node
W20260928 21:57:23.120456 18636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:57:23.121050 18629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:57:23.121388 18629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:57:23.122576 18629 hybrid_clock.cc:648] HybridClock initialized: now 1790632643122544 us; error 47 us; skew 500 ppm
Sep 28 21:57:23 dist-test-slave-k6lz krb5kdc[18083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632643, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:57:23.143911 18629 init.cc:377] Logged in from keytab as kudu/127.2.182.126@KRBTEST.COM (short username kudu)
I20260928 21:57:23.144369 18629 webserver.cc:460] Webserver started at http://127.2.182.126:46749/ using document root <none> and password file <none>
I20260928 21:57:23.144578 18629 fs_manager.cc:359] Metadata directory not provided
I20260928 21:57:23.144630 18629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:57:23.146071 18629 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:57:23.146816 18645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:57:23.147058 18629 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:57:23.147152 18629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/master-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/master-0/wal
uuid: "815c0c5647c04c23a589cb6f6d7e0aff"
format_stamp: "Formatted at 2026-09-28 21:57:02 on dist-test-slave-k6lz"
server_key: "8e03ffd46ff6ac356ad35c6e58bfad73"
server_key_iv: "4be693ad2925a889ba1d8538515d2523"
server_key_version: "encryptionkey@0"
I20260928 21:57:23.147610 18629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:57:23.167295 18629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:57:23.169905 18629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:57:23.170248 18629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:57:23.175073 18629 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.126:42609
I20260928 21:57:23.175180 18697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.126:42609 every 8 connection(s)
I20260928 21:57:23.175546 18629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
I20260928 21:57:23.177096 18698 sys_catalog.cc:263] Verifying existing consensus state
I20260928 21:57:23.177850 18698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 815c0c5647c04c23a589cb6f6d7e0aff: Bootstrap starting.
I20260928 21:57:23.180742 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 18629
I20260928 21:57:23.181142 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.65:38691
--local_ip_for_outbound_sockets=127.2.182.65
--tserver_master_addrs=127.2.182.126:42609
--webserver_port=41235
--webserver_interface=127.2.182.65
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab
--principal=kudu/127.2.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:41569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:57:23.186940 18698 log.cc:824] T 00000000000000000000000000000000 P 815c0c5647c04c23a589cb6f6d7e0aff: Log is configured to *not* fsync() on all Append() calls
I20260928 21:57:23.189074 18698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 815c0c5647c04c23a589cb6f6d7e0aff: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 21:57:23.189386 18698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 815c0c5647c04c23a589cb6f6d7e0aff: Bootstrap complete.
I20260928 21:57:23.191571 18698 raft_consensus.cc:348] T 00000000000000000000000000000000 P 815c0c5647c04c23a589cb6f6d7e0aff [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "815c0c5647c04c23a589cb6f6d7e0aff" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 42609 } }
I20260928 21:57:23.191809 18698 raft_consensus.cc:729] T 00000000000000000000000000000000 P 815c0c5647c04c23a589cb6f6d7e0aff [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 815c0c5647c04c23a589cb6f6d7e0aff, State: Initialized, Role: FOLLOWER
I20260928 21:57:23.191952 18698 consensus_queue.cc:260] T 00000000000000000000000000000000 P 815c0c5647c04c23a589cb6f6d7e0aff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "815c0c5647c04c23a589cb6f6d7e0aff" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 42609 } }
I20260928 21:57:23.192015 18698 raft_consensus.cc:388] T 00000000000000000000000000000000 P 815c0c5647c04c23a589cb6f6d7e0aff [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:57:23.192062 18698 raft_consensus.cc:482] T 00000000000000000000000000000000 P 815c0c5647c04c23a589cb6f6d7e0aff [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:57:23.192109 18698 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 815c0c5647c04c23a589cb6f6d7e0aff [term 1 FOLLOWER]: Advancing to term 2
I20260928 21:57:23.193131 18698 raft_consensus.cc:504] T 00000000000000000000000000000000 P 815c0c5647c04c23a589cb6f6d7e0aff [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "815c0c5647c04c23a589cb6f6d7e0aff" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 42609 } }
I20260928 21:57:23.193222 18698 leader_election.cc:302] T 00000000000000000000000000000000 P 815c0c5647c04c23a589cb6f6d7e0aff [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 815c0c5647c04c23a589cb6f6d7e0aff; no voters:
I20260928 21:57:23.193397 18698 leader_election.cc:288] T 00000000000000000000000000000000 P 815c0c5647c04c23a589cb6f6d7e0aff [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 21:57:23.193460 18702 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 815c0c5647c04c23a589cb6f6d7e0aff [term 2 FOLLOWER]: Leader election won for term 2
I20260928 21:57:23.193583 18702 raft_consensus.cc:686] T 00000000000000000000000000000000 P 815c0c5647c04c23a589cb6f6d7e0aff [term 2 LEADER]: Becoming Leader. State: Replica: 815c0c5647c04c23a589cb6f6d7e0aff, State: Running, Role: LEADER
I20260928 21:57:23.193679 18698 sys_catalog.cc:564] T 00000000000000000000000000000000 P 815c0c5647c04c23a589cb6f6d7e0aff [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:57:23.193742 18702 consensus_queue.cc:237] T 00000000000000000000000000000000 P 815c0c5647c04c23a589cb6f6d7e0aff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "815c0c5647c04c23a589cb6f6d7e0aff" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 42609 } }
I20260928 21:57:23.194031 18702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 815c0c5647c04c23a589cb6f6d7e0aff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 815c0c5647c04c23a589cb6f6d7e0aff. Latest consensus state: current_term: 2 leader_uuid: "815c0c5647c04c23a589cb6f6d7e0aff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "815c0c5647c04c23a589cb6f6d7e0aff" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 42609 } } }
I20260928 21:57:23.194103 18702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 815c0c5647c04c23a589cb6f6d7e0aff [sys.catalog]: This master's current role is: LEADER
I20260928 21:57:23.194252 18704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 815c0c5647c04c23a589cb6f6d7e0aff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "815c0c5647c04c23a589cb6f6d7e0aff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "815c0c5647c04c23a589cb6f6d7e0aff" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 42609 } } }
I20260928 21:57:23.194315 18704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 815c0c5647c04c23a589cb6f6d7e0aff [sys.catalog]: This master's current role is: LEADER
I20260928 21:57:23.194612 18708 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:57:23.195518 18708 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:57:23.195616 18708 catalog_manager.cc:1259] Loaded cluster ID: 5e960ea181ec43699835cf6694426ea5
I20260928 21:57:23.195638 18708 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:57:23.196828 18708 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:57:23.197033 18708 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 815c0c5647c04c23a589cb6f6d7e0aff: Loaded TSK: 0
I20260928 21:57:23.207669 18708 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 815c0c5647c04c23a589cb6f6d7e0aff: Generated new TSK 1
I20260928 21:57:23.207824 18708 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 21:57:23.269546 18700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:57:23.269773 18700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:57:23.269807 18700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:57:23.271129 18700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:57:23.271268 18700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:57:23.271378 18700 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.65
I20260928 21:57:23.273289 18700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:41569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab
--principal=kudu/127.2.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.65:38691
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.182.65
--webserver_port=41235
--tserver_master_addrs=127.2.182.126:42609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18700
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.65
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:57:23.273571 18700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:57:23.273825 18700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:57:23.280186 18726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:57:23.280341 18700 server_base.cc:1034] running on GCE node
W20260928 21:57:23.280414 18728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:57:23.280225 18725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:57:23.280658 18700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:57:23.280915 18700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:57:23.282099 18700 hybrid_clock.cc:648] HybridClock initialized: now 1790632643282085 us; error 27 us; skew 500 ppm
Sep 28 21:57:23 dist-test-slave-k6lz krb5kdc[18083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632643, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:57:23.284847 18700 init.cc:377] Logged in from keytab as kudu/127.2.182.65@KRBTEST.COM (short username kudu)
I20260928 21:57:23.285332 18700 webserver.cc:460] Webserver started at http://127.2.182.65:41235/ using document root <none> and password file <none>
I20260928 21:57:23.285552 18700 fs_manager.cc:359] Metadata directory not provided
I20260928 21:57:23.285606 18700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:57:23.287002 18700 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:57:23.287708 18735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:57:23.287854 18700 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 21:57:23.287937 18700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-0/wal
uuid: "508b43cb26334415a49e0c314db50f3d"
format_stamp: "Formatted at 2026-09-28 21:57:02 on dist-test-slave-k6lz"
server_key: "66e6dfbcdd213691d3dd0eb71920fdf7"
server_key_iv: "95af096eba8faf12a81ed76a744a1830"
server_key_version: "encryptionkey@0"
I20260928 21:57:23.288318 18700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:57:23.302788 18700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:57:23.306172 18700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:57:23.306313 18700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:57:23.306571 18700 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:57:23.307072 18742 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 21:57:23.308614 18700 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 21:57:23.308667 18700 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 21:57:23.308694 18700 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 21:57:23.309324 18700 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 21:57:23.309365 18700 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 21:57:23.309435 18742 tablet_bootstrap.cc:492] T 535438832fe04650857cbf03633e6864 P 508b43cb26334415a49e0c314db50f3d: Bootstrap starting.
I20260928 21:57:23.317234 18700 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.65:38691
I20260928 21:57:23.317418 18849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.65:38691 every 8 connection(s)
I20260928 21:57:23.317772 18700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
I20260928 21:57:23.321337 18742 log.cc:824] T 535438832fe04650857cbf03633e6864 P 508b43cb26334415a49e0c314db50f3d: Log is configured to *not* fsync() on all Append() calls
I20260928 21:57:23.323694 18742 tablet_bootstrap.cc:492] T 535438832fe04650857cbf03633e6864 P 508b43cb26334415a49e0c314db50f3d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 21:57:23.324007 18742 tablet_bootstrap.cc:492] T 535438832fe04650857cbf03633e6864 P 508b43cb26334415a49e0c314db50f3d: Bootstrap complete.
I20260928 21:57:23.324987 18742 ts_tablet_manager.cc:1397] T 535438832fe04650857cbf03633e6864 P 508b43cb26334415a49e0c314db50f3d: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.003s
I20260928 21:57:23.326247 18742 raft_consensus.cc:348] T 535438832fe04650857cbf03633e6864 P 508b43cb26334415a49e0c314db50f3d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bcae726249442b885908d0a151ac987" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 46537 } } peers { permanent_uuid: "d1a41ac432214e4a8b4762b5cf41cce2" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 35109 } } peers { permanent_uuid: "508b43cb26334415a49e0c314db50f3d" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 38691 } }
I20260928 21:57:23.326610 18742 raft_consensus.cc:729] T 535438832fe04650857cbf03633e6864 P 508b43cb26334415a49e0c314db50f3d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 508b43cb26334415a49e0c314db50f3d, State: Initialized, Role: FOLLOWER
I20260928 21:57:23.326838 18742 consensus_queue.cc:260] T 535438832fe04650857cbf03633e6864 P 508b43cb26334415a49e0c314db50f3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bcae726249442b885908d0a151ac987" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 46537 } } peers { permanent_uuid: "d1a41ac432214e4a8b4762b5cf41cce2" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 35109 } } peers { permanent_uuid: "508b43cb26334415a49e0c314db50f3d" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 38691 } }
I20260928 21:57:23.327183 18742 ts_tablet_manager.cc:1428] T 535438832fe04650857cbf03633e6864 P 508b43cb26334415a49e0c314db50f3d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 21:57:23.327183 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 18700
I20260928 21:57:23.327505 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.66:35109
--local_ip_for_outbound_sockets=127.2.182.66
--tserver_master_addrs=127.2.182.126:42609
--webserver_port=32807
--webserver_interface=127.2.182.66
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab
--principal=kudu/127.2.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:41569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 21:57:23 dist-test-slave-k6lz krb5kdc[18083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632643, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.65@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:57:23.332635 18852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:23.318552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:42609 (local address 127.2.182.65:43323)
0928 21:57:23.318934 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:23.318996 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 21:57:23.319725 (+ 729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:57:23.320338 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:57:23.320344 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:57:23.320765 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:57:23.321067 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:57:23.321075 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:57:23.322769 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:57:23.322770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:57:23.324074 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:57:23.324077 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:57:23.324964 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:57:23.324966 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:57:23.325130 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:57:23.326227 (+ 1097us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:57:23.326238 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:57:23.328105 (+ 1867us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:57:23.330539 (+ 2434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:57:23.330550 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:57:23.330552 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:57:23.330925 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:57:23.331281 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:57:23.331282 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:57:23.331282 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:57:23.331452 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:57:23.331978 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:57:23.331985 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:57:23.332142 (+ 157us) client_negotiation.cc:769] Sending connection context
0928 21:57:23.332245 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 21:57:23.332423 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":95,"threads_started":1}
I20260928 21:57:23.333112 18853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:23.318683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.65:43323 (local address 127.2.182.126:42609)
0928 21:57:23.319876 (+ 1193us) server_negotiation.cc:206] Beginning negotiation
0928 21:57:23.319885 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 21:57:23.319931 (+ 46us) server_negotiation.cc:407] Connection header received
0928 21:57:23.320064 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:57:23.320070 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:57:23.320121 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:57:23.320260 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:57:23.321204 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:57:23.322625 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:57:23.324184 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:57:23.324813 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:57:23.324929 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:57:23.328250 (+ 3321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:57:23.328255 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:57:23.328255 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:57:23.328269 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:57:23.330412 (+ 2143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:57:23.331049 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:57:23.331053 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:57:23.331053 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:57:23.331200 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:57:23.331532 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:57:23.331534 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:57:23.331534 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:57:23.331887 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:57:23.331989 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 21:57:23.332345 (+ 356us) server_negotiation.cc:299] Negotiation successful
0928 21:57:23.332536 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1139,"thread_start_us":93,"threads_started":1}
I20260928 21:57:23.333835 18850 heartbeater.cc:347] Connected to a master server at 127.2.182.126:42609
I20260928 21:57:23.333958 18850 heartbeater.cc:464] Registering TS with master...
I20260928 21:57:23.334142 18850 heartbeater.cc:511] Master 127.2.182.126:42609 requested a full tablet report, sending...
I20260928 21:57:23.335000 18662 ts_manager.cc:194] Registered new tserver with Master: 508b43cb26334415a49e0c314db50f3d (127.2.182.65:38691)
I20260928 21:57:23.336298 18662 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.65@KRBTEST.COM'} at 127.2.182.65:43323
I20260928 21:57:23.337376 18850 heartbeater.cc:503] Master 127.2.182.126:42609 was elected leader, sending a full tablet report...
I20260928 21:57:23.343279 18858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:23.335981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53618 (local address 127.2.182.65:38691)
0928 21:57:23.339738 (+ 3757us) server_negotiation.cc:206] Beginning negotiation
0928 21:57:23.339743 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:57:23.339752 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:57:23.339786 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:57:23.339788 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:57:23.339803 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:57:23.339889 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:57:23.340093 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:57:23.340811 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:57:23.342391 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:57:23.342908 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:57:23.342972 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:57:23.343021 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 21:57:23.343181 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 21:57:23.343214 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3723,"thread_start_us":68,"threads_started":1}
I20260928 21:57:23.343312 18857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:23.335720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.65:38691 (local address 127.0.0.1:53618)
0928 21:57:23.339479 (+ 3759us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:23.339527 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 21:57:23.339601 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:57:23.339927 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:57:23.339931 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:57:23.339935 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:57:23.340010 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:57:23.340013 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:57:23.340892 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:57:23.340892 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:57:23.342231 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:57:23.342235 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:57:23.342988 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:57:23.342989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:57:23.343083 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:57:23.343098 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 21:57:23.343167 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 21:57:23.343226 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3720,"thread_start_us":93,"threads_started":1}
I20260928 21:57:23.343852 18784 tablet_service.cc:1514] Processing DeleteTablet for tablet 535438832fe04650857cbf03633e6864 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:57:02 UTC) from {username='kudu', principal='kudu/127.2.182.126@KRBTEST.COM'} at 127.0.0.1:53618
I20260928 21:57:23.344373 18859 tablet_replica.cc:330] stopping tablet replica
I20260928 21:57:23.344477 18859 raft_consensus.cc:2229] T 535438832fe04650857cbf03633e6864 P 508b43cb26334415a49e0c314db50f3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:57:23.344549 18859 raft_consensus.cc:2258] T 535438832fe04650857cbf03633e6864 P 508b43cb26334415a49e0c314db50f3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:57:23.344882 18859 ts_tablet_manager.cc:1905] T 535438832fe04650857cbf03633e6864 P 508b43cb26334415a49e0c314db50f3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:57:23.347229 18859 ts_tablet_manager.cc:1918] T 535438832fe04650857cbf03633e6864 P 508b43cb26334415a49e0c314db50f3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:57:23.347499 18859 log.cc:1196] T 535438832fe04650857cbf03633e6864 P 508b43cb26334415a49e0c314db50f3d: Deleting WAL directory at /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-0/wal/wals/535438832fe04650857cbf03633e6864
I20260928 21:57:23.347713 18859 ts_tablet_manager.cc:1939] T 535438832fe04650857cbf03633e6864 P 508b43cb26334415a49e0c314db50f3d: Deleting consensus metadata
I20260928 21:57:23.348232 18649 catalog_manager.cc:4841] TS 508b43cb26334415a49e0c314db50f3d (127.2.182.65:38691): tablet 535438832fe04650857cbf03633e6864 (table test-table [id=dc59cdfb3a15438db2be21f210d3ab9c]) successfully deleted
W20260928 21:57:23.416646 18856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:57:23.416877 18856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:57:23.416901 18856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:57:23.418157 18856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:57:23.418205 18856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:57:23.418234 18856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.66
I20260928 21:57:23.419791 18856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:41569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab
--principal=kudu/127.2.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.66:35109
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.182.66
--webserver_port=32807
--tserver_master_addrs=127.2.182.126:42609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.66
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:57:23.420034 18856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:57:23.420261 18856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:57:23.425865 18864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:57:23.425884 18865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:57:23.426185 18856 server_base.cc:1034] running on GCE node
W20260928 21:57:23.426260 18867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:57:23.426528 18856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:57:23.426786 18856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:57:23.427948 18856 hybrid_clock.cc:648] HybridClock initialized: now 1790632643427936 us; error 34 us; skew 500 ppm
Sep 28 21:57:23 dist-test-slave-k6lz krb5kdc[18083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632643, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:57:23.430685 18856 init.cc:377] Logged in from keytab as kudu/127.2.182.66@KRBTEST.COM (short username kudu)
I20260928 21:57:23.431195 18856 webserver.cc:460] Webserver started at http://127.2.182.66:32807/ using document root <none> and password file <none>
I20260928 21:57:23.431514 18856 fs_manager.cc:359] Metadata directory not provided
I20260928 21:57:23.431581 18856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:57:23.432966 18856 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:57:23.433665 18874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:57:23.434219 18856 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:57:23.434396 18856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-1/wal
uuid: "d1a41ac432214e4a8b4762b5cf41cce2"
format_stamp: "Formatted at 2026-09-28 21:57:02 on dist-test-slave-k6lz"
server_key: "f23ea8bf03b8d43d9756a4f8f7192561"
server_key_iv: "c509740ca6faa29346854111aaeb97c9"
server_key_version: "encryptionkey@0"
I20260928 21:57:23.434815 18856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:57:23.446436 18856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:57:23.449153 18856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:57:23.449277 18856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:57:23.449491 18856 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:57:23.449944 18856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:57:23.449995 18856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:57:23.450019 18856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:57:23.450030 18856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:57:23.456236 18856 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.66:35109
I20260928 21:57:23.456319 18987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.66:35109 every 8 connection(s)
I20260928 21:57:23.456686 18856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
Sep 28 21:57:23 dist-test-slave-k6lz krb5kdc[18083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632643, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.66@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:57:23.464437 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 18856
I20260928 21:57:23.464732 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.67:46537
--local_ip_for_outbound_sockets=127.2.182.67
--tserver_master_addrs=127.2.182.126:42609
--webserver_port=41839
--webserver_interface=127.2.182.67
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab
--principal=kudu/127.2.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:41569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:57:23.467849 18853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:23.457529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.66:48423 (local address 127.2.182.126:42609)
0928 21:57:23.457583 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 21:57:23.457585 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:57:23.457761 (+ 176us) server_negotiation.cc:407] Connection header received
0928 21:57:23.458277 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:57:23.458278 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:57:23.458292 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:57:23.458421 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:57:23.459555 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:57:23.460114 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:57:23.461297 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:57:23.461496 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:57:23.461567 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:57:23.464327 (+ 2760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:57:23.464329 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:57:23.464329 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:57:23.464338 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:57:23.466025 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:57:23.466694 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:57:23.466696 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:57:23.466696 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:57:23.466770 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:57:23.467092 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:57:23.467093 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:57:23.467094 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:57:23.467246 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:57:23.467308 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 21:57:23.467679 (+ 371us) server_negotiation.cc:299] Negotiation successful
0928 21:57:23.467781 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 21:57:23.468043 18990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:23.457389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:42609 (local address 127.2.182.66:48423)
0928 21:57:23.457583 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:23.457624 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 21:57:23.458155 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:57:23.458494 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:57:23.458498 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:57:23.459081 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:57:23.459294 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:57:23.459299 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:57:23.460216 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:57:23.460217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:57:23.461125 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:57:23.461128 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:57:23.461610 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:57:23.461611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:57:23.461732 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:57:23.462276 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:57:23.462285 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:57:23.464199 (+ 1914us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:57:23.466139 (+ 1940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:57:23.466150 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:57:23.466153 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:57:23.466592 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:57:23.466843 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:57:23.466843 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:57:23.466844 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:57:23.466995 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:57:23.467350 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:57:23.467356 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:57:23.467524 (+ 168us) client_negotiation.cc:769] Sending connection context
0928 21:57:23.467643 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 21:57:23.467814 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":82,"threads_started":1}
I20260928 21:57:23.468719 18988 heartbeater.cc:347] Connected to a master server at 127.2.182.126:42609
I20260928 21:57:23.468820 18988 heartbeater.cc:464] Registering TS with master...
I20260928 21:57:23.469064 18988 heartbeater.cc:511] Master 127.2.182.126:42609 requested a full tablet report, sending...
I20260928 21:57:23.469635 18662 ts_manager.cc:194] Registered new tserver with Master: d1a41ac432214e4a8b4762b5cf41cce2 (127.2.182.66:35109)
I20260928 21:57:23.470208 18662 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.66@KRBTEST.COM'} at 127.2.182.66:48423
W20260928 21:57:23.548326 18991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:57:23.548548 18991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:57:23.548571 18991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:57:23.549795 18991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:57:23.549852 18991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:57:23.549887 18991 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.67
I20260928 21:57:23.551493 18991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:41569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab
--principal=kudu/127.2.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.67:46537
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.182.67
--webserver_port=41839
--tserver_master_addrs=127.2.182.126:42609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18991
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.67
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:57:23.551728 18991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:57:23.551957 18991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:57:23.557421 18997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:57:23.557423 18996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:57:23.557531 18999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:57:23.557578 18991 server_base.cc:1034] running on GCE node
I20260928 21:57:23.557907 18991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:57:23.558135 18991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:57:23.559309 18991 hybrid_clock.cc:648] HybridClock initialized: now 1790632643559240 us; error 87 us; skew 500 ppm
Sep 28 21:57:23 dist-test-slave-k6lz krb5kdc[18083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632643, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:57:23.561774 18991 init.cc:377] Logged in from keytab as kudu/127.2.182.67@KRBTEST.COM (short username kudu)
I20260928 21:57:23.562165 18991 webserver.cc:460] Webserver started at http://127.2.182.67:41839/ using document root <none> and password file <none>
I20260928 21:57:23.562357 18991 fs_manager.cc:359] Metadata directory not provided
I20260928 21:57:23.562404 18991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:57:23.563714 18991 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:57:23.564483 19006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:57:23.564673 18991 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:57:23.564746 18991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-2/wal
uuid: "2bcae726249442b885908d0a151ac987"
format_stamp: "Formatted at 2026-09-28 21:57:02 on dist-test-slave-k6lz"
server_key: "8a7655b4c4a42bdfd9db00aca4487736"
server_key_iv: "de650a7d630e4ad987a8de22b78cdb48"
server_key_version: "encryptionkey@0"
I20260928 21:57:23.565116 18991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:57:23.578989 18991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:57:23.581604 18991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:57:23.581737 18991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:57:23.581941 18991 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:57:23.582206 18991 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:57:23.582235 18991 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:57:23.582257 18991 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:57:23.582267 18991 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:57:23.588183 18991 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.67:46537
I20260928 21:57:23.588224 19119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.67:46537 every 8 connection(s)
I20260928 21:57:23.588555 18991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
I20260928 21:57:23.590692 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 18991
Sep 28 21:57:23 dist-test-slave-k6lz krb5kdc[18083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632643, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.67@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:57:23.599059 19124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:23.591083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32942 (local address 127.2.182.126:42609)
0928 21:57:23.591187 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0928 21:57:23.591188 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:57:23.591279 (+ 91us) server_negotiation.cc:407] Connection header received
0928 21:57:23.591369 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:57:23.591370 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:57:23.591440 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:57:23.591499 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:57:23.592172 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:57:23.592675 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:57:23.593858 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:57:23.594143 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:57:23.594201 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:57:23.595859 (+ 1658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:57:23.595860 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:57:23.595861 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:57:23.595869 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:57:23.597643 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:57:23.598054 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:57:23.598055 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:57:23.598055 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:57:23.598111 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:57:23.598407 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:57:23.598408 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:57:23.598408 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:57:23.598565 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:57:23.598753 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0928 21:57:23.598888 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 21:57:23.598960 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":61,"threads_started":1}
I20260928 21:57:23.599956 18853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:23.589272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.67:36163 (local address 127.2.182.126:42609)
0928 21:57:23.589408 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 21:57:23.589411 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:57:23.589552 (+ 141us) server_negotiation.cc:407] Connection header received
0928 21:57:23.590142 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:57:23.590143 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:57:23.590156 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:57:23.590230 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:57:23.591393 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:57:23.592129 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:57:23.593726 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:57:23.593944 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:57:23.594004 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:57:23.596923 (+ 2919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:57:23.596925 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:57:23.596925 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:57:23.596932 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:57:23.598315 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:57:23.598885 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:57:23.598887 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:57:23.598887 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:57:23.598951 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:57:23.599249 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:57:23.599250 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:57:23.599251 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:57:23.599404 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:57:23.599445 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 21:57:23.599742 (+ 297us) server_negotiation.cc:299] Negotiation successful
0928 21:57:23.599821 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260928 21:57:23.600095 19122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:23.589228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:42609 (local address 127.2.182.67:36163)
0928 21:57:23.589416 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:23.589463 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 21:57:23.590010 (+ 547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:57:23.590308 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:57:23.590313 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:57:23.590978 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:57:23.591265 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:57:23.591273 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:57:23.592428 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:57:23.592429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:57:23.593650 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:57:23.593653 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:57:23.594035 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:57:23.594036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:57:23.594163 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:57:23.594736 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:57:23.594747 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:57:23.596824 (+ 2077us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:57:23.598441 (+ 1617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:57:23.598451 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:57:23.598454 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:57:23.598759 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:57:23.599025 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:57:23.599026 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:57:23.599026 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:57:23.599167 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:57:23.599484 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:57:23.599489 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:57:23.599614 (+ 125us) client_negotiation.cc:769] Sending connection context
0928 21:57:23.599701 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 21:57:23.599868 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":87,"threads_started":1}
I20260928 21:57:23.600505 19120 heartbeater.cc:347] Connected to a master server at 127.2.182.126:42609
I20260928 21:57:23.600606 19120 heartbeater.cc:464] Registering TS with master...
I20260928 21:57:23.600803 19120 heartbeater.cc:511] Master 127.2.182.126:42609 requested a full tablet report, sending...
I20260928 21:57:23.601441 18662 ts_manager.cc:194] Registered new tserver with Master: 2bcae726249442b885908d0a151ac987 (127.2.182.67:46537)
I20260928 21:57:23.601847 18662 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.67@KRBTEST.COM'} at 127.2.182.67:36163
I20260928 21:57:23.611104 2777 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:57:23.614321 18853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:23.611597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32952 (local address 127.2.182.126:42609)
0928 21:57:23.611693 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0928 21:57:23.611695 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:57:23.611857 (+ 162us) server_negotiation.cc:407] Connection header received
0928 21:57:23.611966 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:57:23.611967 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:57:23.612017 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:57:23.612077 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:57:23.612296 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:57:23.612787 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:57:23.613575 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:57:23.613780 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:57:23.613835 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:57:23.614011 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:57:23.614067 (+ 56us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0928 21:57:23.614245 (+ 178us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:32952: token expired
Metrics: {"server-negotiator.queue_time_us":63}
W20260928 21:57:23.614400 18853 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:32952: token expired
W20260928 21:57:23.614545 18591 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.2.182.126:42609)
I20260928 21:57:23.621469 18853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:23.614836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32956 (local address 127.2.182.126:42609)
0928 21:57:23.614884 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 21:57:23.614886 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:57:23.614897 (+ 11us) server_negotiation.cc:407] Connection header received
0928 21:57:23.614924 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:57:23.614925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:57:23.614933 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:57:23.614994 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:57:23.615639 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:57:23.616117 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:57:23.616894 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:57:23.617087 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:57:23.617143 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:57:23.618557 (+ 1414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:57:23.618558 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:57:23.618558 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:57:23.618565 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:57:23.620161 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:57:23.620542 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:57:23.620544 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:57:23.620544 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:57:23.620603 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:57:23.620882 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:57:23.620883 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:57:23.620884 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:57:23.621059 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:57:23.621172 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 21:57:23.621319 (+ 147us) server_negotiation.cc:299] Negotiation successful
0928 21:57:23.621388 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 21:57:23.622787 18662 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32956:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:57:23.624416 18662 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:57:23.627267 18784 tablet_service.cc:1467] Processing CreateTablet for tablet b299fbe4897a41c08b83d662ae37f4af (DEFAULT_TABLE table=test-table [id=14fb378eb3234507851b2e8e246d8466]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:57:23.627600 18784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b299fbe4897a41c08b83d662ae37f4af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:57:23.629182 18742 tablet_bootstrap.cc:492] T b299fbe4897a41c08b83d662ae37f4af P 508b43cb26334415a49e0c314db50f3d: Bootstrap starting.
I20260928 21:57:23.629647 18742 tablet_bootstrap.cc:654] T b299fbe4897a41c08b83d662ae37f4af P 508b43cb26334415a49e0c314db50f3d: Neither blocks nor log segments found. Creating new log.
I20260928 21:57:23.630223 18742 tablet_bootstrap.cc:492] T b299fbe4897a41c08b83d662ae37f4af P 508b43cb26334415a49e0c314db50f3d: No bootstrap required, opened a new log
I20260928 21:57:23.630268 18742 ts_tablet_manager.cc:1397] T b299fbe4897a41c08b83d662ae37f4af P 508b43cb26334415a49e0c314db50f3d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:57:23.630393 18742 raft_consensus.cc:348] T b299fbe4897a41c08b83d662ae37f4af P 508b43cb26334415a49e0c314db50f3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bcae726249442b885908d0a151ac987" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 46537 } } peers { permanent_uuid: "d1a41ac432214e4a8b4762b5cf41cce2" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 35109 } } peers { permanent_uuid: "508b43cb26334415a49e0c314db50f3d" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 38691 } }
I20260928 21:57:23.630468 18742 raft_consensus.cc:374] T b299fbe4897a41c08b83d662ae37f4af P 508b43cb26334415a49e0c314db50f3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:57:23.630524 18742 raft_consensus.cc:729] T b299fbe4897a41c08b83d662ae37f4af P 508b43cb26334415a49e0c314db50f3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 508b43cb26334415a49e0c314db50f3d, State: Initialized, Role: FOLLOWER
I20260928 21:57:23.630568 18742 consensus_queue.cc:260] T b299fbe4897a41c08b83d662ae37f4af P 508b43cb26334415a49e0c314db50f3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bcae726249442b885908d0a151ac987" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 46537 } } peers { permanent_uuid: "d1a41ac432214e4a8b4762b5cf41cce2" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 35109 } } peers { permanent_uuid: "508b43cb26334415a49e0c314db50f3d" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 38691 } }
I20260928 21:57:23.630656 18742 ts_tablet_manager.cc:1428] T b299fbe4897a41c08b83d662ae37f4af P 508b43cb26334415a49e0c314db50f3d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 21:57:23.631788 19127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:23.626759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.66:35109 (local address 127.0.0.1:49724)
0928 21:57:23.627065 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:23.627075 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:57:23.627134 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:57:23.627298 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:57:23.627299 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:57:23.627300 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:57:23.627486 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:57:23.627488 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:57:23.628631 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:57:23.628632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:57:23.630461 (+ 1829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:57:23.630463 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:57:23.631441 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:57:23.631442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:57:23.631507 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:57:23.631513 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 21:57:23.631571 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 21:57:23.631631 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"mutex_wait_us":41,"thread_start_us":163,"threads_started":1}
I20260928 21:57:23.631850 19128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:23.626884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49724 (local address 127.2.182.66:35109)
0928 21:57:23.627047 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 21:57:23.627051 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:57:23.627131 (+ 80us) server_negotiation.cc:407] Connection header received
0928 21:57:23.627198 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:57:23.627200 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:57:23.627215 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:57:23.627282 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:57:23.627573 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:57:23.628531 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:57:23.630547 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:57:23.631305 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:57:23.631434 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:57:23.631500 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 21:57:23.631720 (+ 220us) server_negotiation.cc:299] Negotiation successful
0928 21:57:23.631767 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":86,"threads_started":1}
I20260928 21:57:23.632015 18857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:23.626666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.67:46537 (local address 127.0.0.1:45898)
0928 21:57:23.626735 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:23.626747 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:57:23.626785 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:57:23.627150 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:57:23.627152 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:57:23.627154 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:57:23.627207 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:57:23.627209 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:57:23.628742 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:57:23.628743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:57:23.630265 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:57:23.630351 (+ 86us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:57:23.631707 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:57:23.631708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:57:23.631840 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:57:23.631845 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 21:57:23.631916 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 21:57:23.631966 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":33}
I20260928 21:57:23.632109 19126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:23.626759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45898 (local address 127.2.182.67:46537)
0928 21:57:23.626956 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 21:57:23.626972 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 21:57:23.626983 (+ 11us) server_negotiation.cc:407] Connection header received
0928 21:57:23.627026 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:57:23.627028 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:57:23.627046 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:57:23.627275 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:57:23.627475 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:57:23.628653 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:57:23.630436 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:57:23.631549 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:57:23.631619 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:57:23.631678 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 21:57:23.631963 (+ 285us) server_negotiation.cc:299] Negotiation successful
0928 21:57:23.632011 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":125,"threads_started":1}
I20260928 21:57:23.632903 19054 tablet_service.cc:1467] Processing CreateTablet for tablet b299fbe4897a41c08b83d662ae37f4af (DEFAULT_TABLE table=test-table [id=14fb378eb3234507851b2e8e246d8466]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:57:23.632897 18922 tablet_service.cc:1467] Processing CreateTablet for tablet b299fbe4897a41c08b83d662ae37f4af (DEFAULT_TABLE table=test-table [id=14fb378eb3234507851b2e8e246d8466]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:57:23.633298 19054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b299fbe4897a41c08b83d662ae37f4af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:57:23.633298 18922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b299fbe4897a41c08b83d662ae37f4af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:57:23.637033 19131 tablet_bootstrap.cc:492] T b299fbe4897a41c08b83d662ae37f4af P d1a41ac432214e4a8b4762b5cf41cce2: Bootstrap starting.
I20260928 21:57:23.637722 19131 tablet_bootstrap.cc:654] T b299fbe4897a41c08b83d662ae37f4af P d1a41ac432214e4a8b4762b5cf41cce2: Neither blocks nor log segments found. Creating new log.
I20260928 21:57:23.638247 19131 log.cc:824] T b299fbe4897a41c08b83d662ae37f4af P d1a41ac432214e4a8b4762b5cf41cce2: Log is configured to *not* fsync() on all Append() calls
I20260928 21:57:23.638688 19130 tablet_bootstrap.cc:492] T b299fbe4897a41c08b83d662ae37f4af P 2bcae726249442b885908d0a151ac987: Bootstrap starting.
I20260928 21:57:23.639084 19131 tablet_bootstrap.cc:492] T b299fbe4897a41c08b83d662ae37f4af P d1a41ac432214e4a8b4762b5cf41cce2: No bootstrap required, opened a new log
I20260928 21:57:23.639158 19131 ts_tablet_manager.cc:1397] T b299fbe4897a41c08b83d662ae37f4af P d1a41ac432214e4a8b4762b5cf41cce2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 21:57:23.639395 19130 tablet_bootstrap.cc:654] T b299fbe4897a41c08b83d662ae37f4af P 2bcae726249442b885908d0a151ac987: Neither blocks nor log segments found. Creating new log.
I20260928 21:57:23.640021 19130 log.cc:824] T b299fbe4897a41c08b83d662ae37f4af P 2bcae726249442b885908d0a151ac987: Log is configured to *not* fsync() on all Append() calls
I20260928 21:57:23.640816 19130 tablet_bootstrap.cc:492] T b299fbe4897a41c08b83d662ae37f4af P 2bcae726249442b885908d0a151ac987: No bootstrap required, opened a new log
I20260928 21:57:23.640902 19130 ts_tablet_manager.cc:1397] T b299fbe4897a41c08b83d662ae37f4af P 2bcae726249442b885908d0a151ac987: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 21:57:23.641170 19131 raft_consensus.cc:348] T b299fbe4897a41c08b83d662ae37f4af P d1a41ac432214e4a8b4762b5cf41cce2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bcae726249442b885908d0a151ac987" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 46537 } } peers { permanent_uuid: "d1a41ac432214e4a8b4762b5cf41cce2" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 35109 } } peers { permanent_uuid: "508b43cb26334415a49e0c314db50f3d" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 38691 } }
I20260928 21:57:23.641368 19131 raft_consensus.cc:374] T b299fbe4897a41c08b83d662ae37f4af P d1a41ac432214e4a8b4762b5cf41cce2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:57:23.641404 19131 raft_consensus.cc:729] T b299fbe4897a41c08b83d662ae37f4af P d1a41ac432214e4a8b4762b5cf41cce2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a41ac432214e4a8b4762b5cf41cce2, State: Initialized, Role: FOLLOWER
I20260928 21:57:23.641517 19131 consensus_queue.cc:260] T b299fbe4897a41c08b83d662ae37f4af P d1a41ac432214e4a8b4762b5cf41cce2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bcae726249442b885908d0a151ac987" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 46537 } } peers { permanent_uuid: "d1a41ac432214e4a8b4762b5cf41cce2" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 35109 } } peers { permanent_uuid: "508b43cb26334415a49e0c314db50f3d" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 38691 } }
I20260928 21:57:23.641829 19131 ts_tablet_manager.cc:1428] T b299fbe4897a41c08b83d662ae37f4af P d1a41ac432214e4a8b4762b5cf41cce2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 21:57:23.641930 18988 heartbeater.cc:503] Master 127.2.182.126:42609 was elected leader, sending a full tablet report...
I20260928 21:57:23.643177 19130 raft_consensus.cc:348] T b299fbe4897a41c08b83d662ae37f4af P 2bcae726249442b885908d0a151ac987 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bcae726249442b885908d0a151ac987" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 46537 } } peers { permanent_uuid: "d1a41ac432214e4a8b4762b5cf41cce2" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 35109 } } peers { permanent_uuid: "508b43cb26334415a49e0c314db50f3d" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 38691 } }
I20260928 21:57:23.643483 19130 raft_consensus.cc:374] T b299fbe4897a41c08b83d662ae37f4af P 2bcae726249442b885908d0a151ac987 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:57:23.643527 19130 raft_consensus.cc:729] T b299fbe4897a41c08b83d662ae37f4af P 2bcae726249442b885908d0a151ac987 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bcae726249442b885908d0a151ac987, State: Initialized, Role: FOLLOWER
I20260928 21:57:23.643623 19130 consensus_queue.cc:260] T b299fbe4897a41c08b83d662ae37f4af P 2bcae726249442b885908d0a151ac987 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bcae726249442b885908d0a151ac987" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 46537 } } peers { permanent_uuid: "d1a41ac432214e4a8b4762b5cf41cce2" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 35109 } } peers { permanent_uuid: "508b43cb26334415a49e0c314db50f3d" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 38691 } }
I20260928 21:57:23.644006 19130 ts_tablet_manager.cc:1428] T b299fbe4897a41c08b83d662ae37f4af P 2bcae726249442b885908d0a151ac987: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
I20260928 21:57:23.644083 19120 heartbeater.cc:503] Master 127.2.182.126:42609 was elected leader, sending a full tablet report...
I20260928 21:57:23.688323 18855 raft_consensus.cc:482] T b299fbe4897a41c08b83d662ae37f4af P 508b43cb26334415a49e0c314db50f3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:57:23.688472 18855 raft_consensus.cc:504] T b299fbe4897a41c08b83d662ae37f4af P 508b43cb26334415a49e0c314db50f3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bcae726249442b885908d0a151ac987" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 46537 } } peers { permanent_uuid: "d1a41ac432214e4a8b4762b5cf41cce2" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 35109 } } peers { permanent_uuid: "508b43cb26334415a49e0c314db50f3d" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 38691 } }
I20260928 21:57:23.688844 18855 leader_election.cc:288] T b299fbe4897a41c08b83d662ae37f4af P 508b43cb26334415a49e0c314db50f3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2bcae726249442b885908d0a151ac987 (127.2.182.67:46537), d1a41ac432214e4a8b4762b5cf41cce2 (127.2.182.66:35109)
I20260928 21:57:23.692533 19128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:23.689112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.65:55313 (local address 127.2.182.66:35109)
0928 21:57:23.689152 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 21:57:23.689154 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:57:23.689292 (+ 138us) server_negotiation.cc:407] Connection header received
0928 21:57:23.689400 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:57:23.689402 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:57:23.689411 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:57:23.689464 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:57:23.689639 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:57:23.690202 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:57:23.691569 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:57:23.692182 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:57:23.692246 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:57:23.692254 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 21:57:23.692417 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 21:57:23.692460 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 21:57:23.692581 19126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:23.689108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.65:50969 (local address 127.2.182.67:46537)
0928 21:57:23.689152 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 21:57:23.689154 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:57:23.689164 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:57:23.689199 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:57:23.689200 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:57:23.689209 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:57:23.689261 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:57:23.689424 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:57:23.690026 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:57:23.691516 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:57:23.692114 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:57:23.692198 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:57:23.692208 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 21:57:23.692472 (+ 264us) server_negotiation.cc:299] Negotiation successful
0928 21:57:23.692523 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 21:57:23.692562 19136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:23.689036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.66:35109 (local address 127.2.182.65:55313)
0928 21:57:23.689232 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:23.689241 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 21:57:23.689319 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:57:23.689472 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:57:23.689473 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:57:23.689475 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:57:23.689553 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:57:23.689554 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:57:23.690312 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:57:23.690313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:57:23.691473 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:57:23.691475 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:57:23.692282 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:57:23.692283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:57:23.692336 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:57:23.692339 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:57:23.692403 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 21:57:23.692449 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":112,"threads_started":1}
I20260928 21:57:23.692562 18852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:23.689019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.67:46537 (local address 127.2.182.65:50969)
0928 21:57:23.689088 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:23.689100 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:57:23.689151 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:57:23.689269 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:57:23.689270 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:57:23.689272 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:57:23.689345 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:57:23.689347 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:57:23.690113 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:57:23.690115 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:57:23.691418 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:57:23.691421 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:57:23.692234 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:57:23.692235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:57:23.692327 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:57:23.692332 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 21:57:23.692460 (+ 128us) client_negotiation.cc:240] Negotiation successful
0928 21:57:23.692500 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 21:57:23.693099 19074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b299fbe4897a41c08b83d662ae37f4af" candidate_uuid: "508b43cb26334415a49e0c314db50f3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bcae726249442b885908d0a151ac987" is_pre_election: true
I20260928 21:57:23.693084 18942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b299fbe4897a41c08b83d662ae37f4af" candidate_uuid: "508b43cb26334415a49e0c314db50f3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a41ac432214e4a8b4762b5cf41cce2" is_pre_election: true
I20260928 21:57:23.693269 18942 raft_consensus.cc:2454] T b299fbe4897a41c08b83d662ae37f4af P d1a41ac432214e4a8b4762b5cf41cce2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 508b43cb26334415a49e0c314db50f3d in term 0.
I20260928 21:57:23.693267 19074 raft_consensus.cc:2454] T b299fbe4897a41c08b83d662ae37f4af P 2bcae726249442b885908d0a151ac987 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 508b43cb26334415a49e0c314db50f3d in term 0.
I20260928 21:57:23.693459 18739 leader_election.cc:302] T b299fbe4897a41c08b83d662ae37f4af P 508b43cb26334415a49e0c314db50f3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 508b43cb26334415a49e0c314db50f3d, d1a41ac432214e4a8b4762b5cf41cce2; no voters:
I20260928 21:57:23.693570 18855 raft_consensus.cc:2781] T b299fbe4897a41c08b83d662ae37f4af P 508b43cb26334415a49e0c314db50f3d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:57:23.693629 18855 raft_consensus.cc:482] T b299fbe4897a41c08b83d662ae37f4af P 508b43cb26334415a49e0c314db50f3d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:57:23.693645 18855 raft_consensus.cc:3037] T b299fbe4897a41c08b83d662ae37f4af P 508b43cb26334415a49e0c314db50f3d [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:57:23.694458 18855 raft_consensus.cc:504] T b299fbe4897a41c08b83d662ae37f4af P 508b43cb26334415a49e0c314db50f3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bcae726249442b885908d0a151ac987" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 46537 } } peers { permanent_uuid: "d1a41ac432214e4a8b4762b5cf41cce2" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 35109 } } peers { permanent_uuid: "508b43cb26334415a49e0c314db50f3d" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 38691 } }
I20260928 21:57:23.694588 18855 leader_election.cc:288] T b299fbe4897a41c08b83d662ae37f4af P 508b43cb26334415a49e0c314db50f3d [CANDIDATE]: Term 1 election: Requested vote from peers 2bcae726249442b885908d0a151ac987 (127.2.182.67:46537), d1a41ac432214e4a8b4762b5cf41cce2 (127.2.182.66:35109)
I20260928 21:57:23.694816 19074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b299fbe4897a41c08b83d662ae37f4af" candidate_uuid: "508b43cb26334415a49e0c314db50f3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bcae726249442b885908d0a151ac987"
I20260928 21:57:23.694860 18942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b299fbe4897a41c08b83d662ae37f4af" candidate_uuid: "508b43cb26334415a49e0c314db50f3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a41ac432214e4a8b4762b5cf41cce2"
I20260928 21:57:23.694926 19074 raft_consensus.cc:3037] T b299fbe4897a41c08b83d662ae37f4af P 2bcae726249442b885908d0a151ac987 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:57:23.694949 18942 raft_consensus.cc:3037] T b299fbe4897a41c08b83d662ae37f4af P d1a41ac432214e4a8b4762b5cf41cce2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:57:23.695688 18942 raft_consensus.cc:2454] T b299fbe4897a41c08b83d662ae37f4af P d1a41ac432214e4a8b4762b5cf41cce2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 508b43cb26334415a49e0c314db50f3d in term 1.
I20260928 21:57:23.695779 19074 raft_consensus.cc:2454] T b299fbe4897a41c08b83d662ae37f4af P 2bcae726249442b885908d0a151ac987 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 508b43cb26334415a49e0c314db50f3d in term 1.
I20260928 21:57:23.695904 18739 leader_election.cc:302] T b299fbe4897a41c08b83d662ae37f4af P 508b43cb26334415a49e0c314db50f3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 508b43cb26334415a49e0c314db50f3d, d1a41ac432214e4a8b4762b5cf41cce2; no voters:
I20260928 21:57:23.696015 18855 raft_consensus.cc:2781] T b299fbe4897a41c08b83d662ae37f4af P 508b43cb26334415a49e0c314db50f3d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:57:23.696151 18855 raft_consensus.cc:686] T b299fbe4897a41c08b83d662ae37f4af P 508b43cb26334415a49e0c314db50f3d [term 1 LEADER]: Becoming Leader. State: Replica: 508b43cb26334415a49e0c314db50f3d, State: Running, Role: LEADER
I20260928 21:57:23.696204 18855 consensus_queue.cc:237] T b299fbe4897a41c08b83d662ae37f4af P 508b43cb26334415a49e0c314db50f3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bcae726249442b885908d0a151ac987" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 46537 } } peers { permanent_uuid: "d1a41ac432214e4a8b4762b5cf41cce2" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 35109 } } peers { permanent_uuid: "508b43cb26334415a49e0c314db50f3d" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 38691 } }
I20260928 21:57:23.697100 18662 catalog_manager.cc:5495] T b299fbe4897a41c08b83d662ae37f4af P 508b43cb26334415a49e0c314db50f3d reported cstate change: term changed from 0 to 1, leader changed from <none> to 508b43cb26334415a49e0c314db50f3d (127.2.182.65). New cstate: current_term: 1 leader_uuid: "508b43cb26334415a49e0c314db50f3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bcae726249442b885908d0a151ac987" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 46537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1a41ac432214e4a8b4762b5cf41cce2" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 35109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "508b43cb26334415a49e0c314db50f3d" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 38691 } health_report { overall_health: HEALTHY } } }
W20260928 21:57:23.707402 18989 tablet.cc:2369] T b299fbe4897a41c08b83d662ae37f4af P d1a41ac432214e4a8b4762b5cf41cce2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:57:23.709981 18858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:23.707221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53622 (local address 127.2.182.65:38691)
0928 21:57:23.707379 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 21:57:23.707381 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:57:23.707392 (+ 11us) server_negotiation.cc:407] Connection header received
0928 21:57:23.707433 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:57:23.707434 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:57:23.707457 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:57:23.707536 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:57:23.707757 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:57:23.708310 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:57:23.709131 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:57:23.709391 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:57:23.709452 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:57:23.709621 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:57:23.709740 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:57:23.709814 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 21:57:23.709899 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 21:57:23.709923 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
I20260928 21:57:23.711633 18942 raft_consensus.cc:1261] T b299fbe4897a41c08b83d662ae37f4af P d1a41ac432214e4a8b4762b5cf41cce2 [term 1 FOLLOWER]: Refusing update from remote peer 508b43cb26334415a49e0c314db50f3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:57:23.711634 19074 raft_consensus.cc:1261] T b299fbe4897a41c08b83d662ae37f4af P 2bcae726249442b885908d0a151ac987 [term 1 FOLLOWER]: Refusing update from remote peer 508b43cb26334415a49e0c314db50f3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:57:23.711972 19137 consensus_queue.cc:1035] T b299fbe4897a41c08b83d662ae37f4af P 508b43cb26334415a49e0c314db50f3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a41ac432214e4a8b4762b5cf41cce2" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 35109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:57:23.712109 18855 consensus_queue.cc:1035] T b299fbe4897a41c08b83d662ae37f4af P 508b43cb26334415a49e0c314db50f3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bcae726249442b885908d0a151ac987" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 46537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:57:23.714834 19141 mvcc.cc:205] Tried to move back new op lower bound from 7334431308640276480 to 7334431308580175872. Current Snapshot: MvccSnapshot[applied={T|T < 7334431308640276480}]
I20260928 21:57:23.714968 19140 mvcc.cc:205] Tried to move back new op lower bound from 7334431308640276480 to 7334431308580175872. Current Snapshot: MvccSnapshot[applied={T|T < 7334431308640276480}]
I20260928 21:57:23.717480 18661 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:57:23.717588 18661 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:57:23.718385 18661 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 815c0c5647c04c23a589cb6f6d7e0aff: Sending DeleteTablet for 3 replicas of tablet b299fbe4897a41c08b83d662ae37f4af
I20260928 21:57:23.718731 18922 tablet_service.cc:1514] Processing DeleteTablet for tablet b299fbe4897a41c08b83d662ae37f4af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:57:23 UTC) from {username='kudu', principal='kudu/127.2.182.126@KRBTEST.COM'} at 127.0.0.1:49724
I20260928 21:57:23.718981 19054 tablet_service.cc:1514] Processing DeleteTablet for tablet b299fbe4897a41c08b83d662ae37f4af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:57:23 UTC) from {username='kudu', principal='kudu/127.2.182.126@KRBTEST.COM'} at 127.0.0.1:45898
I20260928 21:57:23.718932 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 18700
I20260928 21:57:23.723523 19151 tablet_replica.cc:330] T b299fbe4897a41c08b83d662ae37f4af P 2bcae726249442b885908d0a151ac987: stopping tablet replica
I20260928 21:57:23.723701 19151 raft_consensus.cc:2229] T b299fbe4897a41c08b83d662ae37f4af P 2bcae726249442b885908d0a151ac987 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:57:23.723598 19150 tablet_replica.cc:330] T b299fbe4897a41c08b83d662ae37f4af P d1a41ac432214e4a8b4762b5cf41cce2: stopping tablet replica
I20260928 21:57:23.723851 19151 raft_consensus.cc:2258] T b299fbe4897a41c08b83d662ae37f4af P 2bcae726249442b885908d0a151ac987 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:57:23.723938 19150 raft_consensus.cc:2229] T b299fbe4897a41c08b83d662ae37f4af P d1a41ac432214e4a8b4762b5cf41cce2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:57:23.724113 19150 raft_consensus.cc:2258] T b299fbe4897a41c08b83d662ae37f4af P d1a41ac432214e4a8b4762b5cf41cce2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:57:23.724699 19151 ts_tablet_manager.cc:1905] T b299fbe4897a41c08b83d662ae37f4af P 2bcae726249442b885908d0a151ac987: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:57:23.724836 19150 ts_tablet_manager.cc:1905] T b299fbe4897a41c08b83d662ae37f4af P d1a41ac432214e4a8b4762b5cf41cce2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 21:57:23.725343 18649 connection.cc:535] client connection to 127.2.182.65:38691 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 21:57:23.725476 18649 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 21:57:23.725590 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 18856
I20260928 21:57:23.727821 18857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:23.727685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.65:38691 (local address 127.0.0.1:53634)
0928 21:57:23.727738 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:23.727785 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.65:38691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 21:57:23.727949 18649 catalog_manager.cc:4584] TS 508b43cb26334415a49e0c314db50f3d (127.2.182.65:38691): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b299fbe4897a41c08b83d662ae37f4af: Network error: Client connection negotiation failed: client connection to 127.2.182.65:38691: connect: Connection refused (error 111)
I20260928 21:57:23.728546 19151 ts_tablet_manager.cc:1918] T b299fbe4897a41c08b83d662ae37f4af P 2bcae726249442b885908d0a151ac987: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:57:23.728679 19151 log.cc:1196] T b299fbe4897a41c08b83d662ae37f4af P 2bcae726249442b885908d0a151ac987: Deleting WAL directory at /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-2/wal/wals/b299fbe4897a41c08b83d662ae37f4af
I20260928 21:57:23.728957 19151 ts_tablet_manager.cc:1939] T b299fbe4897a41c08b83d662ae37f4af P 2bcae726249442b885908d0a151ac987: Deleting consensus metadata
I20260928 21:57:23.729538 18647 catalog_manager.cc:4841] TS 2bcae726249442b885908d0a151ac987 (127.2.182.67:46537): tablet b299fbe4897a41c08b83d662ae37f4af (table test-table [id=14fb378eb3234507851b2e8e246d8466]) successfully deleted
I20260928 21:57:23.730494 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 18991
I20260928 21:57:23.730613 18857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:23.730503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.66:35109 (local address 127.0.0.1:49730)
0928 21:57:23.730552 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:23.730580 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.66:35109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 21:57:23.735297 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 18629
I20260928 21:57:43.740428 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:42609
--webserver_interface=127.2.182.126
--webserver_port=46749
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab
--principal=kudu/127.2.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:41569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.126:42609
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:57:43.826678 19153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:57:43.826915 19153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:57:43.826938 19153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:57:43.828202 19153 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 21:57:43.828248 19153 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 21:57:43.828260 19153 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 21:57:43.828270 19153 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:57:43.828282 19153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:57:43.828294 19153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:57:43.828305 19153 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:57:43.828317 19153 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:57:43.829910 19153 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:41569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.182.126:42609
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab
--principal=kudu/127.2.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:42609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.126
--webserver_port=46749
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:57:43.830152 19153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:57:43.830379 19153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:57:43.835476 19159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:57:43.835589 19158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:57:43.835644 19153 server_base.cc:1034] running on GCE node
W20260928 21:57:43.835472 19161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:57:43.835933 19153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:57:43.836249 19153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:57:43.837412 19153 hybrid_clock.cc:648] HybridClock initialized: now 1790632663837388 us; error 54 us; skew 500 ppm
Sep 28 21:57:43 dist-test-slave-k6lz krb5kdc[18083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632663, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:57:43.846642 19153 init.cc:377] Logged in from keytab as kudu/127.2.182.126@KRBTEST.COM (short username kudu)
I20260928 21:57:43.847091 19153 webserver.cc:460] Webserver started at http://127.2.182.126:46749/ using document root <none> and password file <none>
I20260928 21:57:43.847296 19153 fs_manager.cc:359] Metadata directory not provided
I20260928 21:57:43.847374 19153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:57:43.848861 19153 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:57:43.849989 19168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:57:43.850207 19153 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:57:43.850298 19153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/master-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/master-0/wal
uuid: "815c0c5647c04c23a589cb6f6d7e0aff"
format_stamp: "Formatted at 2026-09-28 21:57:02 on dist-test-slave-k6lz"
server_key: "8e03ffd46ff6ac356ad35c6e58bfad73"
server_key_iv: "4be693ad2925a889ba1d8538515d2523"
server_key_version: "encryptionkey@0"
I20260928 21:57:43.850736 19153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:57:43.864892 19153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:57:43.867215 19153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:57:43.867473 19153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:57:43.871887 19153 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.126:42609
I20260928 21:57:43.871948 19220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.126:42609 every 8 connection(s)
I20260928 21:57:43.872282 19153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
I20260928 21:57:43.873769 19221 sys_catalog.cc:263] Verifying existing consensus state
I20260928 21:57:43.874408 19221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 815c0c5647c04c23a589cb6f6d7e0aff: Bootstrap starting.
I20260928 21:57:43.875906 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 19153
I20260928 21:57:43.876268 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.65:38691
--local_ip_for_outbound_sockets=127.2.182.65
--tserver_master_addrs=127.2.182.126:42609
--webserver_port=41235
--webserver_interface=127.2.182.65
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab
--principal=kudu/127.2.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:41569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:57:43.883383 19221 log.cc:824] T 00000000000000000000000000000000 P 815c0c5647c04c23a589cb6f6d7e0aff: Log is configured to *not* fsync() on all Append() calls
I20260928 21:57:43.885794 19221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 815c0c5647c04c23a589cb6f6d7e0aff: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 21:57:43.886085 19221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 815c0c5647c04c23a589cb6f6d7e0aff: Bootstrap complete.
I20260928 21:57:43.888273 19221 raft_consensus.cc:348] T 00000000000000000000000000000000 P 815c0c5647c04c23a589cb6f6d7e0aff [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "815c0c5647c04c23a589cb6f6d7e0aff" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 42609 } }
I20260928 21:57:43.888517 19221 raft_consensus.cc:729] T 00000000000000000000000000000000 P 815c0c5647c04c23a589cb6f6d7e0aff [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 815c0c5647c04c23a589cb6f6d7e0aff, State: Initialized, Role: FOLLOWER
I20260928 21:57:43.888666 19221 consensus_queue.cc:260] T 00000000000000000000000000000000 P 815c0c5647c04c23a589cb6f6d7e0aff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "815c0c5647c04c23a589cb6f6d7e0aff" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 42609 } }
I20260928 21:57:43.888743 19221 raft_consensus.cc:388] T 00000000000000000000000000000000 P 815c0c5647c04c23a589cb6f6d7e0aff [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:57:43.888774 19221 raft_consensus.cc:482] T 00000000000000000000000000000000 P 815c0c5647c04c23a589cb6f6d7e0aff [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:57:43.888819 19221 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 815c0c5647c04c23a589cb6f6d7e0aff [term 2 FOLLOWER]: Advancing to term 3
I20260928 21:57:43.889858 19221 raft_consensus.cc:504] T 00000000000000000000000000000000 P 815c0c5647c04c23a589cb6f6d7e0aff [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "815c0c5647c04c23a589cb6f6d7e0aff" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 42609 } }
I20260928 21:57:43.889963 19221 leader_election.cc:302] T 00000000000000000000000000000000 P 815c0c5647c04c23a589cb6f6d7e0aff [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 815c0c5647c04c23a589cb6f6d7e0aff; no voters:
I20260928 21:57:43.890131 19221 leader_election.cc:288] T 00000000000000000000000000000000 P 815c0c5647c04c23a589cb6f6d7e0aff [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 21:57:43.890182 19225 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 815c0c5647c04c23a589cb6f6d7e0aff [term 3 FOLLOWER]: Leader election won for term 3
I20260928 21:57:43.890306 19225 raft_consensus.cc:686] T 00000000000000000000000000000000 P 815c0c5647c04c23a589cb6f6d7e0aff [term 3 LEADER]: Becoming Leader. State: Replica: 815c0c5647c04c23a589cb6f6d7e0aff, State: Running, Role: LEADER
I20260928 21:57:43.890404 19221 sys_catalog.cc:564] T 00000000000000000000000000000000 P 815c0c5647c04c23a589cb6f6d7e0aff [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:57:43.890393 19225 consensus_queue.cc:237] T 00000000000000000000000000000000 P 815c0c5647c04c23a589cb6f6d7e0aff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "815c0c5647c04c23a589cb6f6d7e0aff" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 42609 } }
I20260928 21:57:43.890728 19225 sys_catalog.cc:455] T 00000000000000000000000000000000 P 815c0c5647c04c23a589cb6f6d7e0aff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 815c0c5647c04c23a589cb6f6d7e0aff. Latest consensus state: current_term: 3 leader_uuid: "815c0c5647c04c23a589cb6f6d7e0aff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "815c0c5647c04c23a589cb6f6d7e0aff" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 42609 } } }
I20260928 21:57:43.890832 19225 sys_catalog.cc:458] T 00000000000000000000000000000000 P 815c0c5647c04c23a589cb6f6d7e0aff [sys.catalog]: This master's current role is: LEADER
I20260928 21:57:43.891078 19227 sys_catalog.cc:455] T 00000000000000000000000000000000 P 815c0c5647c04c23a589cb6f6d7e0aff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "815c0c5647c04c23a589cb6f6d7e0aff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "815c0c5647c04c23a589cb6f6d7e0aff" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 42609 } } }
I20260928 21:57:43.891184 19227 sys_catalog.cc:458] T 00000000000000000000000000000000 P 815c0c5647c04c23a589cb6f6d7e0aff [sys.catalog]: This master's current role is: LEADER
I20260928 21:57:43.891211 19231 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:57:43.892220 19231 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:57:43.892344 19231 catalog_manager.cc:1259] Loaded cluster ID: 5e960ea181ec43699835cf6694426ea5
I20260928 21:57:43.892671 19231 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:57:43.893494 19231 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:57:43.893656 19231 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 815c0c5647c04c23a589cb6f6d7e0aff: Loaded TSK: 0
I20260928 21:57:43.893695 19231 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 815c0c5647c04c23a589cb6f6d7e0aff: Loaded TSK: 1
F20260928 21:57:43.893903 19231 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fe98202a9ad google::LogMessage::Fail() at ??:0
@ 0x7fe98202d88a google::LogMessage::SendToLog() at ??:0
@ 0x7fe98202a4de google::LogMessage::Flush() at ??:0
@ 0x7fe98202ba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fe984797104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fe984794d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fe989f841cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fe989f8448e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fe989f84594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fe989f4ac48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fe989f54ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fe983ef80d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fe983eed954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe984d516db start_thread at ??:0
@ 0x7fe980b2861f clone at ??:0
W20260928 21:57:43.979605 19223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:57:43.979804 19223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:57:43.979830 19223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:57:43.981014 19223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:57:43.981057 19223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:57:43.981083 19223 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.65
I20260928 21:57:43.982685 19223 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:41569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab
--principal=kudu/127.2.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.65:38691
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.182.65
--webserver_port=41235
--tserver_master_addrs=127.2.182.126:42609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19223
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.65
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:57:43.982932 19223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:57:43.983165 19223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:57:43.988421 19265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:57:43.988513 19263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:57:43.988440 19262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:57:43.988768 19223 server_base.cc:1034] running on GCE node
I20260928 21:57:43.988981 19223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:57:43.989252 19223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:57:43.990419 19223 hybrid_clock.cc:648] HybridClock initialized: now 1790632663990405 us; error 28 us; skew 500 ppm
Sep 28 21:57:43 dist-test-slave-k6lz krb5kdc[18083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632663, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:57:43.993533 19223 init.cc:377] Logged in from keytab as kudu/127.2.182.65@KRBTEST.COM (short username kudu)
I20260928 21:57:43.994115 19223 webserver.cc:460] Webserver started at http://127.2.182.65:41235/ using document root <none> and password file <none>
I20260928 21:57:43.994405 19223 fs_manager.cc:359] Metadata directory not provided
I20260928 21:57:43.994468 19223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:57:43.996258 19223 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:57:43.997063 19272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:57:43.997241 19223 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:57:43.997335 19223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-0/wal
uuid: "508b43cb26334415a49e0c314db50f3d"
format_stamp: "Formatted at 2026-09-28 21:57:02 on dist-test-slave-k6lz"
server_key: "66e6dfbcdd213691d3dd0eb71920fdf7"
server_key_iv: "95af096eba8faf12a81ed76a744a1830"
server_key_version: "encryptionkey@0"
I20260928 21:57:43.997668 19223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:57:44.013115 19223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:57:44.015479 19223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:57:44.015622 19223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:57:44.015872 19223 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:57:44.016419 19279 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 21:57:44.018390 19223 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 21:57:44.018451 19223 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 21:57:44.018486 19223 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 21:57:44.019366 19223 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 21:57:44.019409 19223 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.001s
I20260928 21:57:44.019474 19279 tablet_bootstrap.cc:492] T b299fbe4897a41c08b83d662ae37f4af P 508b43cb26334415a49e0c314db50f3d: Bootstrap starting.
I20260928 21:57:44.026398 19223 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.65:38691
I20260928 21:57:44.026499 19386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.65:38691 every 8 connection(s)
I20260928 21:57:44.026798 19223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
I20260928 21:57:44.029806 19279 log.cc:824] T b299fbe4897a41c08b83d662ae37f4af P 508b43cb26334415a49e0c314db50f3d: Log is configured to *not* fsync() on all Append() calls
I20260928 21:57:44.032024 19279 tablet_bootstrap.cc:492] T b299fbe4897a41c08b83d662ae37f4af P 508b43cb26334415a49e0c314db50f3d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 21:57:44.032019 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 19223
I20260928 21:57:44.032290 19279 tablet_bootstrap.cc:492] T b299fbe4897a41c08b83d662ae37f4af P 508b43cb26334415a49e0c314db50f3d: Bootstrap complete.
I20260928 21:57:44.032282 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.66:35109
--local_ip_for_outbound_sockets=127.2.182.66
--tserver_master_addrs=127.2.182.126:42609
--webserver_port=32807
--webserver_interface=127.2.182.66
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab
--principal=kudu/127.2.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:41569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:57:44.033092 19279 ts_tablet_manager.cc:1397] T b299fbe4897a41c08b83d662ae37f4af P 508b43cb26334415a49e0c314db50f3d: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.006s
I20260928 21:57:44.034101 19279 raft_consensus.cc:348] T b299fbe4897a41c08b83d662ae37f4af P 508b43cb26334415a49e0c314db50f3d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bcae726249442b885908d0a151ac987" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 46537 } } peers { permanent_uuid: "d1a41ac432214e4a8b4762b5cf41cce2" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 35109 } } peers { permanent_uuid: "508b43cb26334415a49e0c314db50f3d" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 38691 } }
I20260928 21:57:44.034346 19279 raft_consensus.cc:729] T b299fbe4897a41c08b83d662ae37f4af P 508b43cb26334415a49e0c314db50f3d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 508b43cb26334415a49e0c314db50f3d, State: Initialized, Role: FOLLOWER
I20260928 21:57:44.034487 19279 consensus_queue.cc:260] T b299fbe4897a41c08b83d662ae37f4af P 508b43cb26334415a49e0c314db50f3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bcae726249442b885908d0a151ac987" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 46537 } } peers { permanent_uuid: "d1a41ac432214e4a8b4762b5cf41cce2" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 35109 } } peers { permanent_uuid: "508b43cb26334415a49e0c314db50f3d" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 38691 } }
I20260928 21:57:44.034734 19279 ts_tablet_manager.cc:1428] T b299fbe4897a41c08b83d662ae37f4af P 508b43cb26334415a49e0c314db50f3d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
W20260928 21:57:44.124693 19389 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:57:44.027497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:42609 (local address 127.2.182.65:48729)
0928 21:57:44.030134 (+ 2637us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:44.030185 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 21:57:44.030847 (+ 662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:57:44.124509 (+ 93662us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:42609: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":2592,"thread_start_us":69,"threads_started":1}
W20260928 21:57:44.125020 19387 heartbeater.cc:644] Failed to heartbeat to 127.2.182.126:42609 (0 consecutive failures): Network error: Failed to ping master at 127.2.182.126:42609: Client connection negotiation failed: client connection to 127.2.182.126:42609: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 21:57:44.125339 19389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:44.125228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:42609 (local address 127.2.182.65:40975)
0928 21:57:44.125275 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:44.125314 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:42609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 21:57:44.125644 19389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:44.125563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:42609 (local address 127.2.182.65:59161)
0928 21:57:44.125604 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:44.125627 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:42609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 21:57:44.125743 19387 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 21:57:44.144119 19391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:57:44.144343 19391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:57:44.144377 19391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:57:44.145602 19391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:57:44.145648 19391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:57:44.145685 19391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.66
I20260928 21:57:44.147169 19391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:41569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab
--principal=kudu/127.2.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.66:35109
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.182.66
--webserver_port=32807
--tserver_master_addrs=127.2.182.126:42609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19391
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.66
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:57:44.147432 19391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:57:44.147650 19391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:57:44.152787 19397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:57:44.152796 19398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:57:44.153052 19391 server_base.cc:1034] running on GCE node
W20260928 21:57:44.153086 19400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:57:44.153262 19391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:57:44.153504 19391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:57:44.154665 19391 hybrid_clock.cc:648] HybridClock initialized: now 1790632664154651 us; error 30 us; skew 500 ppm
Sep 28 21:57:44 dist-test-slave-k6lz krb5kdc[18083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632664, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:57:44.157261 19391 init.cc:377] Logged in from keytab as kudu/127.2.182.66@KRBTEST.COM (short username kudu)
I20260928 21:57:44.157714 19391 webserver.cc:460] Webserver started at http://127.2.182.66:32807/ using document root <none> and password file <none>
I20260928 21:57:44.157910 19391 fs_manager.cc:359] Metadata directory not provided
I20260928 21:57:44.157953 19391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:57:44.159312 19391 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:57:44.160280 19407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:57:44.160549 19391 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:57:44.160640 19391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-1/wal
uuid: "d1a41ac432214e4a8b4762b5cf41cce2"
format_stamp: "Formatted at 2026-09-28 21:57:02 on dist-test-slave-k6lz"
server_key: "f23ea8bf03b8d43d9756a4f8f7192561"
server_key_iv: "c509740ca6faa29346854111aaeb97c9"
server_key_version: "encryptionkey@0"
I20260928 21:57:44.161039 19391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:57:44.178174 19391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:57:44.180619 19391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:57:44.180742 19391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:57:44.180979 19391 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:57:44.181465 19414 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 21:57:44.183048 19391 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 21:57:44.183106 19391 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 21:57:44.183144 19391 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 21:57:44.183858 19391 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 21:57:44.183892 19391 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 21:57:44.183976 19414 tablet_bootstrap.cc:492] T b299fbe4897a41c08b83d662ae37f4af P d1a41ac432214e4a8b4762b5cf41cce2: Bootstrap starting.
I20260928 21:57:44.185748 19414 log.cc:824] T b299fbe4897a41c08b83d662ae37f4af P d1a41ac432214e4a8b4762b5cf41cce2: Log is configured to *not* fsync() on all Append() calls
I20260928 21:57:44.188174 19414 tablet_bootstrap.cc:492] T b299fbe4897a41c08b83d662ae37f4af P d1a41ac432214e4a8b4762b5cf41cce2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 21:57:44.188457 19414 tablet_bootstrap.cc:492] T b299fbe4897a41c08b83d662ae37f4af P d1a41ac432214e4a8b4762b5cf41cce2: Bootstrap complete.
I20260928 21:57:44.188618 19414 ts_tablet_manager.cc:1397] T b299fbe4897a41c08b83d662ae37f4af P d1a41ac432214e4a8b4762b5cf41cce2: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 21:57:44.189913 19414 raft_consensus.cc:348] T b299fbe4897a41c08b83d662ae37f4af P d1a41ac432214e4a8b4762b5cf41cce2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bcae726249442b885908d0a151ac987" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 46537 } } peers { permanent_uuid: "d1a41ac432214e4a8b4762b5cf41cce2" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 35109 } } peers { permanent_uuid: "508b43cb26334415a49e0c314db50f3d" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 38691 } }
I20260928 21:57:44.190151 19414 raft_consensus.cc:729] T b299fbe4897a41c08b83d662ae37f4af P d1a41ac432214e4a8b4762b5cf41cce2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a41ac432214e4a8b4762b5cf41cce2, State: Initialized, Role: FOLLOWER
I20260928 21:57:44.190315 19414 consensus_queue.cc:260] T b299fbe4897a41c08b83d662ae37f4af P d1a41ac432214e4a8b4762b5cf41cce2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bcae726249442b885908d0a151ac987" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 46537 } } peers { permanent_uuid: "d1a41ac432214e4a8b4762b5cf41cce2" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 35109 } } peers { permanent_uuid: "508b43cb26334415a49e0c314db50f3d" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 38691 } }
I20260928 21:57:44.190582 19391 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.66:35109
I20260928 21:57:44.190593 19414 ts_tablet_manager.cc:1428] T b299fbe4897a41c08b83d662ae37f4af P d1a41ac432214e4a8b4762b5cf41cce2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 21:57:44.190625 19523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.66:35109 every 8 connection(s)
I20260928 21:57:44.190932 19391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
I20260928 21:57:44.192049 19526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:44.191624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:42609 (local address 127.2.182.66:40797)
0928 21:57:44.191823 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:44.191934 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:42609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":69,"threads_started":1}
W20260928 21:57:44.192229 19524 heartbeater.cc:644] Failed to heartbeat to 127.2.182.126:42609 (0 consecutive failures): Network error: Failed to ping master at 127.2.182.126:42609: Client connection negotiation failed: client connection to 127.2.182.126:42609: connect: Connection refused (error 111)
I20260928 21:57:44.192543 19526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:44.192429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:42609 (local address 127.2.182.66:54711)
0928 21:57:44.192492 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:44.192521 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:42609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 21:57:44.192764 19526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:44.192713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:42609 (local address 127.2.182.66:42119)
0928 21:57:44.192732 (+ 19us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:44.192750 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:42609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260928 21:57:44.192855 19524 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 21:57:44.198866 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 19391
I20260928 21:57:44.199105 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.67:46537
--local_ip_for_outbound_sockets=127.2.182.67
--tserver_master_addrs=127.2.182.126:42609
--webserver_port=41839
--webserver_interface=127.2.182.67
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab
--principal=kudu/127.2.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:41569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:57:44.280246 19527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:57:44.280464 19527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:57:44.280488 19527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:57:44.281669 19527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:57:44.281718 19527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:57:44.281745 19527 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.67
I20260928 21:57:44.283435 19527 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:41569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab
--principal=kudu/127.2.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.67:46537
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.182.67
--webserver_port=41839
--tserver_master_addrs=127.2.182.126:42609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19527
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.67
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:57:44.283684 19527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:57:44.283912 19527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:57:44.289004 19535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:57:44.288995 19533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:57:44.288995 19532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:57:44.289884 19527 server_base.cc:1034] running on GCE node
I20260928 21:57:44.290119 19527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:57:44.290351 19527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:57:44.291489 19527 hybrid_clock.cc:648] HybridClock initialized: now 1790632664291452 us; error 51 us; skew 500 ppm
Sep 28 21:57:44 dist-test-slave-k6lz krb5kdc[18083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632664, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:57:44.294031 19527 init.cc:377] Logged in from keytab as kudu/127.2.182.67@KRBTEST.COM (short username kudu)
I20260928 21:57:44.294509 19527 webserver.cc:460] Webserver started at http://127.2.182.67:41839/ using document root <none> and password file <none>
I20260928 21:57:44.294723 19527 fs_manager.cc:359] Metadata directory not provided
I20260928 21:57:44.294766 19527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:57:44.296068 19527 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260928 21:57:44.296620 19542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:57:44.296761 19527 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 21:57:44.296839 19527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-2/wal
uuid: "2bcae726249442b885908d0a151ac987"
format_stamp: "Formatted at 2026-09-28 21:57:02 on dist-test-slave-k6lz"
server_key: "8a7655b4c4a42bdfd9db00aca4487736"
server_key_iv: "de650a7d630e4ad987a8de22b78cdb48"
server_key_version: "encryptionkey@0"
I20260928 21:57:44.297210 19527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:57:44.308019 19527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:57:44.310264 19527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:57:44.310407 19527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:57:44.310627 19527 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:57:44.310925 19527 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:57:44.310956 19527 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:57:44.310985 19527 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:57:44.311009 19527 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:57:44.317287 19527 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.67:46537
I20260928 21:57:44.317386 19655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.67:46537 every 8 connection(s)
I20260928 21:57:44.317770 19527 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
I20260928 21:57:44.318866 19658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:44.318483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:42609 (local address 127.2.182.67:37805)
0928 21:57:44.318648 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:44.318720 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:42609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":76,"threads_started":1}
W20260928 21:57:44.319238 19656 heartbeater.cc:644] Failed to heartbeat to 127.2.182.126:42609 (0 consecutive failures): Network error: Failed to ping master at 127.2.182.126:42609: Client connection negotiation failed: client connection to 127.2.182.126:42609: connect: Connection refused (error 111)
I20260928 21:57:44.319638 19658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:44.319537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:42609 (local address 127.2.182.67:54341)
0928 21:57:44.319585 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:44.319616 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:42609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 21:57:44.319971 19658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:44.319859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:42609 (local address 127.2.182.67:37107)
0928 21:57:44.319919 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:44.319948 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:42609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 21:57:44.320147 19656 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 21:57:44.324143 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 19527
I20260928 21:57:45.126514 19660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:45.126150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:42609 (local address 127.2.182.65:39287)
0928 21:57:45.126311 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:45.126385 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:42609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":76,"threads_started":1}
I20260928 21:57:45.193367 19661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:45.193177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:42609 (local address 127.2.182.66:60117)
0928 21:57:45.193285 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:45.193323 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:42609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":52,"threads_started":1}
I20260928 21:57:45.271961 19662 raft_consensus.cc:482] T b299fbe4897a41c08b83d662ae37f4af P 508b43cb26334415a49e0c314db50f3d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:57:45.272086 19662 raft_consensus.cc:504] T b299fbe4897a41c08b83d662ae37f4af P 508b43cb26334415a49e0c314db50f3d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bcae726249442b885908d0a151ac987" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 46537 } } peers { permanent_uuid: "d1a41ac432214e4a8b4762b5cf41cce2" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 35109 } } peers { permanent_uuid: "508b43cb26334415a49e0c314db50f3d" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 38691 } }
I20260928 21:57:45.272428 19662 leader_election.cc:288] T b299fbe4897a41c08b83d662ae37f4af P 508b43cb26334415a49e0c314db50f3d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2bcae726249442b885908d0a151ac987 (127.2.182.67:46537), d1a41ac432214e4a8b4762b5cf41cce2 (127.2.182.66:35109)
Sep 28 21:57:45 dist-test-slave-k6lz krb5kdc[18083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632663, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.65@KRBTEST.COM for kudu/127.2.182.67@KRBTEST.COM
Sep 28 21:57:45 dist-test-slave-k6lz krb5kdc[18083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632663, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.65@KRBTEST.COM for kudu/127.2.182.66@KRBTEST.COM
I20260928 21:57:45.282529 19664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:45.272668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.65:47667 (local address 127.2.182.67:46537)
0928 21:57:45.272801 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 21:57:45.272812 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 21:57:45.272836 (+ 24us) server_negotiation.cc:407] Connection header received
0928 21:57:45.273345 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:57:45.273347 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:57:45.273410 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:57:45.273536 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:57:45.273865 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:57:45.274896 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:57:45.275989 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:57:45.276383 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:57:45.276486 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:57:45.278838 (+ 2352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:57:45.278843 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:57:45.278844 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:57:45.278856 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:57:45.280774 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:57:45.281286 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:57:45.281289 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:57:45.281290 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:57:45.281388 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:57:45.281619 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:57:45.281619 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:57:45.281620 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:57:45.281885 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:57:45.281956 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 21:57:45.282344 (+ 388us) server_negotiation.cc:299] Negotiation successful
0928 21:57:45.282456 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":59,"threads_started":1}
I20260928 21:57:45.282538 19660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:45.272560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.67:46537 (local address 127.2.182.65:47667)
0928 21:57:45.272607 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:45.272617 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:57:45.272667 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:57:45.273537 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:57:45.273538 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:57:45.273689 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:57:45.273757 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:57:45.273763 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:57:45.275007 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:57:45.275008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:57:45.275899 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:57:45.275902 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:57:45.276466 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:57:45.276467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:57:45.276573 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:57:45.277089 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:57:45.277097 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:57:45.278709 (+ 1612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:57:45.280885 (+ 2176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:57:45.280894 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:57:45.280897 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:57:45.281208 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:57:45.281448 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:57:45.281449 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:57:45.281449 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:57:45.281561 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:57:45.281949 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:57:45.281956 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:57:45.282106 (+ 150us) client_negotiation.cc:769] Sending connection context
0928 21:57:45.282214 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 21:57:45.282421 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":26}
I20260928 21:57:45.282593 19663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:45.272560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.66:35109 (local address 127.2.182.65:35711)
0928 21:57:45.272698 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:45.272706 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 21:57:45.272745 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:57:45.273166 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:57:45.273170 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:57:45.273550 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:57:45.273757 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:57:45.273763 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:57:45.275177 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:57:45.275178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:57:45.275905 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:57:45.275908 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:57:45.276463 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:57:45.276464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:57:45.276574 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:57:45.277131 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:57:45.277135 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:57:45.279095 (+ 1960us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:57:45.280902 (+ 1807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:57:45.280903 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:57:45.280904 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:57:45.281224 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:57:45.281513 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:57:45.281514 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:57:45.281514 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:57:45.281572 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:57:45.282009 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:57:45.282010 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:57:45.282106 (+ 96us) client_negotiation.cc:769] Sending connection context
0928 21:57:45.282214 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 21:57:45.282487 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":129,"thread_start_us":65,"threads_started":1}
I20260928 21:57:45.282966 19665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:45.272691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.65:35711 (local address 127.2.182.66:35109)
0928 21:57:45.272833 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 21:57:45.272841 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 21:57:45.272862 (+ 21us) server_negotiation.cc:407] Connection header received
0928 21:57:45.272952 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:57:45.272953 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:57:45.272991 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:57:45.273101 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:57:45.273897 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:57:45.275082 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:57:45.275988 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:57:45.276377 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:57:45.276476 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:57:45.279272 (+ 2796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:57:45.279276 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:57:45.279276 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:57:45.279289 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:57:45.280814 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:57:45.281323 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:57:45.281327 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:57:45.281327 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:57:45.281442 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:57:45.281626 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:57:45.281626 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:57:45.281627 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:57:45.281922 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:57:45.282018 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 21:57:45.282713 (+ 695us) server_negotiation.cc:299] Negotiation successful
0928 21:57:45.282870 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":44,"threads_started":1}
W20260928 21:57:45.283380 19274 leader_election.cc:341] T b299fbe4897a41c08b83d662ae37f4af P 508b43cb26334415a49e0c314db50f3d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2bcae726249442b885908d0a151ac987 (127.2.182.67:46537): Not found: Tablet not found: b299fbe4897a41c08b83d662ae37f4af
I20260928 21:57:45.283567 19477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b299fbe4897a41c08b83d662ae37f4af" candidate_uuid: "508b43cb26334415a49e0c314db50f3d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d1a41ac432214e4a8b4762b5cf41cce2" is_pre_election: true
I20260928 21:57:45.283700 19477 raft_consensus.cc:2454] T b299fbe4897a41c08b83d662ae37f4af P d1a41ac432214e4a8b4762b5cf41cce2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 508b43cb26334415a49e0c314db50f3d in term 1.
I20260928 21:57:45.283984 19276 leader_election.cc:302] T b299fbe4897a41c08b83d662ae37f4af P 508b43cb26334415a49e0c314db50f3d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 508b43cb26334415a49e0c314db50f3d, d1a41ac432214e4a8b4762b5cf41cce2; no voters: 2bcae726249442b885908d0a151ac987
I20260928 21:57:45.284096 19662 raft_consensus.cc:2781] T b299fbe4897a41c08b83d662ae37f4af P 508b43cb26334415a49e0c314db50f3d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 21:57:45.284157 19662 raft_consensus.cc:482] T b299fbe4897a41c08b83d662ae37f4af P 508b43cb26334415a49e0c314db50f3d [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:57:45.284185 19662 raft_consensus.cc:3037] T b299fbe4897a41c08b83d662ae37f4af P 508b43cb26334415a49e0c314db50f3d [term 1 FOLLOWER]: Advancing to term 2
I20260928 21:57:45.285386 19662 raft_consensus.cc:504] T b299fbe4897a41c08b83d662ae37f4af P 508b43cb26334415a49e0c314db50f3d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bcae726249442b885908d0a151ac987" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 46537 } } peers { permanent_uuid: "d1a41ac432214e4a8b4762b5cf41cce2" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 35109 } } peers { permanent_uuid: "508b43cb26334415a49e0c314db50f3d" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 38691 } }
I20260928 21:57:45.285518 19662 leader_election.cc:288] T b299fbe4897a41c08b83d662ae37f4af P 508b43cb26334415a49e0c314db50f3d [CANDIDATE]: Term 2 election: Requested vote from peers 2bcae726249442b885908d0a151ac987 (127.2.182.67:46537), d1a41ac432214e4a8b4762b5cf41cce2 (127.2.182.66:35109)
I20260928 21:57:45.285744 19477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b299fbe4897a41c08b83d662ae37f4af" candidate_uuid: "508b43cb26334415a49e0c314db50f3d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d1a41ac432214e4a8b4762b5cf41cce2"
I20260928 21:57:45.285832 19477 raft_consensus.cc:3037] T b299fbe4897a41c08b83d662ae37f4af P d1a41ac432214e4a8b4762b5cf41cce2 [term 1 FOLLOWER]: Advancing to term 2
W20260928 21:57:45.285858 19274 leader_election.cc:341] T b299fbe4897a41c08b83d662ae37f4af P 508b43cb26334415a49e0c314db50f3d [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 2bcae726249442b885908d0a151ac987 (127.2.182.67:46537): Not found: Tablet not found: b299fbe4897a41c08b83d662ae37f4af
I20260928 21:57:45.287041 19477 raft_consensus.cc:2454] T b299fbe4897a41c08b83d662ae37f4af P d1a41ac432214e4a8b4762b5cf41cce2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 508b43cb26334415a49e0c314db50f3d in term 2.
I20260928 21:57:45.287276 19276 leader_election.cc:302] T b299fbe4897a41c08b83d662ae37f4af P 508b43cb26334415a49e0c314db50f3d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 508b43cb26334415a49e0c314db50f3d, d1a41ac432214e4a8b4762b5cf41cce2; no voters: 2bcae726249442b885908d0a151ac987
I20260928 21:57:45.287458 19662 raft_consensus.cc:2781] T b299fbe4897a41c08b83d662ae37f4af P 508b43cb26334415a49e0c314db50f3d [term 2 FOLLOWER]: Leader election won for term 2
I20260928 21:57:45.287608 19662 raft_consensus.cc:686] T b299fbe4897a41c08b83d662ae37f4af P 508b43cb26334415a49e0c314db50f3d [term 2 LEADER]: Becoming Leader. State: Replica: 508b43cb26334415a49e0c314db50f3d, State: Running, Role: LEADER
I20260928 21:57:45.287693 19662 consensus_queue.cc:237] T b299fbe4897a41c08b83d662ae37f4af P 508b43cb26334415a49e0c314db50f3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bcae726249442b885908d0a151ac987" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 46537 } } peers { permanent_uuid: "d1a41ac432214e4a8b4762b5cf41cce2" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 35109 } } peers { permanent_uuid: "508b43cb26334415a49e0c314db50f3d" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 38691 } }
I20260928 21:57:45.287976 19660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:45.287860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:42609 (local address 127.2.182.65:40877)
0928 21:57:45.287911 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:45.287946 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:42609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 21:57:45.320853 19668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:45.320570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:42609 (local address 127.2.182.67:57901)
0928 21:57:45.320718 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:45.320769 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:42609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":73,"threads_started":1}
I20260928 21:57:45.810541 19477 raft_consensus.cc:1261] T b299fbe4897a41c08b83d662ae37f4af P d1a41ac432214e4a8b4762b5cf41cce2 [term 2 FOLLOWER]: Refusing update from remote peer 508b43cb26334415a49e0c314db50f3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260928 21:57:45.810933 19669 consensus_queue.cc:1035] T b299fbe4897a41c08b83d662ae37f4af P 508b43cb26334415a49e0c314db50f3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a41ac432214e4a8b4762b5cf41cce2" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 35109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260928 21:57:45.811756 19672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:45.811608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:42609 (local address 127.2.182.66:48455)
0928 21:57:45.811688 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:45.811723 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:42609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69,"thread_start_us":41,"threads_started":1}
I20260928 21:57:45.812846 19677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:45.812636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:42609 (local address 127.2.182.65:51901)
0928 21:57:45.812769 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:45.812806 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:42609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":85,"threads_started":1}
W20260928 21:57:45.812868 19274 consensus_peers.cc:487] T b299fbe4897a41c08b83d662ae37f4af P 508b43cb26334415a49e0c314db50f3d -> Peer 2bcae726249442b885908d0a151ac987 (127.2.182.67:46537): Couldn't send request to peer 2bcae726249442b885908d0a151ac987. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b299fbe4897a41c08b83d662ae37f4af. This is attempt 1: this message will repeat every 5th retry.
I20260928 21:57:45.812995 19672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:45.812903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:42609 (local address 127.2.182.66:55451)
0928 21:57:45.812941 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:45.812970 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:42609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260928 21:57:46.321497 19678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:46.321282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:42609 (local address 127.2.182.67:56701)
0928 21:57:46.321404 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:46.321441 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:42609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":58,"threads_started":1}
I20260928 21:57:46.398994 19680 ts_tablet_manager.cc:927] T b299fbe4897a41c08b83d662ae37f4af P 2bcae726249442b885908d0a151ac987: Initiating tablet copy from peer 508b43cb26334415a49e0c314db50f3d (127.2.182.65:38691)
I20260928 21:57:46.399212 19680 tablet_copy_client.cc:316] T b299fbe4897a41c08b83d662ae37f4af P 2bcae726249442b885908d0a151ac987: tablet copy: Beginning tablet copy session from remote peer at address 127.2.182.65:38691
Sep 28 21:57:46 dist-test-slave-k6lz krb5kdc[18083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632664, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.67@KRBTEST.COM for kudu/127.2.182.65@KRBTEST.COM
I20260928 21:57:46.407433 19678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:46.399504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.65:38691 (local address 127.2.182.67:48201)
0928 21:57:46.399548 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:46.399568 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 21:57:46.399659 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:57:46.399850 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:57:46.399854 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:57:46.400144 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:57:46.400205 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:57:46.400208 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:57:46.401143 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:57:46.401144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:57:46.401816 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:57:46.401819 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:57:46.402262 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:57:46.402263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:57:46.402353 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:57:46.402825 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:57:46.402829 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:57:46.404487 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:57:46.406234 (+ 1747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:57:46.406241 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:57:46.406243 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:57:46.406433 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:57:46.406635 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:57:46.406636 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:57:46.406636 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:57:46.406711 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:57:46.407048 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:57:46.407055 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:57:46.407102 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 21:57:46.407172 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 21:57:46.407298 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260928 21:57:46.407521 19682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:46.399596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.67:48201 (local address 127.2.182.65:38691)
0928 21:57:46.399723 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 21:57:46.399728 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:57:46.399737 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:57:46.399762 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:57:46.399764 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:57:46.399778 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:57:46.399846 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:57:46.400284 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:57:46.401061 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:57:46.401899 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:57:46.402199 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:57:46.402256 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:57:46.404575 (+ 2319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:57:46.404583 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:57:46.404584 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:57:46.404592 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:57:46.406132 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:57:46.406517 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:57:46.406520 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:57:46.406520 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:57:46.406579 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:57:46.406764 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:57:46.406765 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:57:46.406765 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:57:46.406935 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:57:46.407006 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 21:57:46.407216 (+ 210us) server_negotiation.cc:299] Negotiation successful
0928 21:57:46.407435 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":62,"threads_started":1}
I20260928 21:57:46.408061 19361 tablet_copy_service.cc:140] P 508b43cb26334415a49e0c314db50f3d: Received BeginTabletCopySession request for tablet b299fbe4897a41c08b83d662ae37f4af from peer 2bcae726249442b885908d0a151ac987 ({username='kudu', principal='kudu/127.2.182.67@KRBTEST.COM'} at 127.2.182.67:48201)
I20260928 21:57:46.408141 19361 tablet_copy_service.cc:161] P 508b43cb26334415a49e0c314db50f3d: Beginning new tablet copy session on tablet b299fbe4897a41c08b83d662ae37f4af from peer 2bcae726249442b885908d0a151ac987 at {username='kudu', principal='kudu/127.2.182.67@KRBTEST.COM'} at 127.2.182.67:48201: session id = 2bcae726249442b885908d0a151ac987-b299fbe4897a41c08b83d662ae37f4af
I20260928 21:57:46.408906 19361 tablet_copy_source_session.cc:205] T b299fbe4897a41c08b83d662ae37f4af P 508b43cb26334415a49e0c314db50f3d: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 21:57:46.409967 19680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b299fbe4897a41c08b83d662ae37f4af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:57:46.412758 19680 tablet_copy_client.cc:791] T b299fbe4897a41c08b83d662ae37f4af P 2bcae726249442b885908d0a151ac987: tablet copy: Starting download of 0 data blocks...
I20260928 21:57:46.412889 19680 tablet_copy_client.cc:655] T b299fbe4897a41c08b83d662ae37f4af P 2bcae726249442b885908d0a151ac987: tablet copy: Starting download of 1 WAL segments...
I20260928 21:57:46.413622 19680 tablet_copy_client.cc:531] T b299fbe4897a41c08b83d662ae37f4af P 2bcae726249442b885908d0a151ac987: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 21:57:46.414271 19680 tablet_bootstrap.cc:492] T b299fbe4897a41c08b83d662ae37f4af P 2bcae726249442b885908d0a151ac987: Bootstrap starting.
I20260928 21:57:46.415923 19680 log.cc:824] T b299fbe4897a41c08b83d662ae37f4af P 2bcae726249442b885908d0a151ac987: Log is configured to *not* fsync() on all Append() calls
I20260928 21:57:46.417889 19680 tablet_bootstrap.cc:492] T b299fbe4897a41c08b83d662ae37f4af P 2bcae726249442b885908d0a151ac987: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 21:57:46.418087 19680 tablet_bootstrap.cc:492] T b299fbe4897a41c08b83d662ae37f4af P 2bcae726249442b885908d0a151ac987: Bootstrap complete.
I20260928 21:57:46.418202 19680 ts_tablet_manager.cc:1397] T b299fbe4897a41c08b83d662ae37f4af P 2bcae726249442b885908d0a151ac987: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260928 21:57:46.419162 19680 raft_consensus.cc:348] T b299fbe4897a41c08b83d662ae37f4af P 2bcae726249442b885908d0a151ac987 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bcae726249442b885908d0a151ac987" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 46537 } } peers { permanent_uuid: "d1a41ac432214e4a8b4762b5cf41cce2" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 35109 } } peers { permanent_uuid: "508b43cb26334415a49e0c314db50f3d" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 38691 } }
I20260928 21:57:46.419318 19680 raft_consensus.cc:729] T b299fbe4897a41c08b83d662ae37f4af P 2bcae726249442b885908d0a151ac987 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bcae726249442b885908d0a151ac987, State: Initialized, Role: FOLLOWER
I20260928 21:57:46.419484 19680 consensus_queue.cc:260] T b299fbe4897a41c08b83d662ae37f4af P 2bcae726249442b885908d0a151ac987 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bcae726249442b885908d0a151ac987" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 46537 } } peers { permanent_uuid: "d1a41ac432214e4a8b4762b5cf41cce2" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 35109 } } peers { permanent_uuid: "508b43cb26334415a49e0c314db50f3d" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 38691 } }
I20260928 21:57:46.419728 19680 ts_tablet_manager.cc:1428] T b299fbe4897a41c08b83d662ae37f4af P 2bcae726249442b885908d0a151ac987: Time spent starting tablet: real 0.001s user 0.000s sys 0.003s
I20260928 21:57:46.420035 19678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:46.419934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:42609 (local address 127.2.182.67:37335)
0928 21:57:46.419980 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:46.420012 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:42609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260928 21:57:46.420027 19361 tablet_copy_service.cc:342] P 508b43cb26334415a49e0c314db50f3d: Request end of tablet copy session 2bcae726249442b885908d0a151ac987-b299fbe4897a41c08b83d662ae37f4af received from {username='kudu', principal='kudu/127.2.182.67@KRBTEST.COM'} at 127.2.182.67:48201
I20260928 21:57:46.420079 19361 tablet_copy_service.cc:433] P 508b43cb26334415a49e0c314db50f3d: ending tablet copy session 2bcae726249442b885908d0a151ac987-b299fbe4897a41c08b83d662ae37f4af on tablet b299fbe4897a41c08b83d662ae37f4af with peer 2bcae726249442b885908d0a151ac987
I20260928 21:57:46.420300 19678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:46.420228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:42609 (local address 127.2.182.67:59983)
0928 21:57:46.420256 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:46.420278 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:42609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
I20260928 21:57:46.813546 19685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:46.813264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:42609 (local address 127.2.182.65:42725)
0928 21:57:46.813430 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:46.813495 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:42609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":74,"threads_started":1}
I20260928 21:57:46.813547 19686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:46.813336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:42609 (local address 127.2.182.66:52197)
0928 21:57:46.813457 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:46.813491 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:42609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":69,"threads_started":1}
I20260928 21:57:47.001803 19685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:47.001624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:42609 (local address 127.2.182.65:45975)
0928 21:57:47.001698 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:47.001738 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:42609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 21:57:47.001803 19610 raft_consensus.cc:1203] T b299fbe4897a41c08b83d662ae37f4af P 2bcae726249442b885908d0a151ac987 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260928 21:57:47.002437 19691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:47.002240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:42609 (local address 127.2.182.67:48283)
0928 21:57:47.002352 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:47.002386 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:42609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":44,"threads_started":1}
I20260928 21:57:47.814500 19693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:47.814130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:42609 (local address 127.2.182.66:60129)
0928 21:57:47.814317 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:47.814369 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:42609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":69,"threads_started":1}
I20260928 21:57:48.002527 19694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:48.002293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:42609 (local address 127.2.182.65:55445)
0928 21:57:48.002437 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:48.002480 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:42609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":64,"threads_started":1}
I20260928 21:57:48.003014 19695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:48.002820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:42609 (local address 127.2.182.67:49391)
0928 21:57:48.002940 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:48.002977 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:42609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":68,"threads_started":1}
I20260928 21:57:48.815258 19696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:48.815030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:42609 (local address 127.2.182.66:34677)
0928 21:57:48.815171 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:48.815212 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:42609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":60,"threads_started":1}
I20260928 21:57:49.003255 19697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:49.003009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:42609 (local address 127.2.182.65:33575)
0928 21:57:49.003168 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:49.003210 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:42609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":65,"threads_started":1}
I20260928 21:57:49.003628 19698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:49.003452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:42609 (local address 127.2.182.67:39897)
0928 21:57:49.003553 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:49.003590 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:42609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":52,"threads_started":1}
I20260928 21:57:49.816121 19699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:49.815890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:42609 (local address 127.2.182.66:43619)
0928 21:57:49.816030 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:49.816075 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:42609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":63,"threads_started":1}
I20260928 21:57:50.004235 19700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:50.003976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:42609 (local address 127.2.182.65:45227)
0928 21:57:50.004140 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:50.004188 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:42609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":75,"threads_started":1}
I20260928 21:57:50.004259 19701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:50.004076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:42609 (local address 127.2.182.67:43361)
0928 21:57:50.004176 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:50.004217 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:42609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":48,"threads_started":1}
I20260928 21:57:50.004714 19700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:50.004624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:42609 (local address 127.2.182.65:48901)
0928 21:57:50.004667 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:50.004692 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:42609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 21:57:50.816998 19703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:50.816729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:42609 (local address 127.2.182.66:60509)
0928 21:57:50.816875 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:50.816949 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:42609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":65,"threads_started":1}
I20260928 21:57:51.005147 19704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:51.004881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:42609 (local address 127.2.182.67:44113)
0928 21:57:51.005046 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:51.005099 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:42609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":81,"threads_started":1}
I20260928 21:57:51.005275 19705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:51.005077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:42609 (local address 127.2.182.65:57401)
0928 21:57:51.005192 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:51.005224 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:42609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":54,"threads_started":1}
I20260928 21:57:51.817793 19706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:51.817569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:42609 (local address 127.2.182.66:58243)
0928 21:57:51.817707 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:51.817745 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:42609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":63,"threads_started":1}
I20260928 21:57:52.005911 19707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:52.005638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:42609 (local address 127.2.182.67:41727)
0928 21:57:52.005791 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:52.005825 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:42609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":80,"threads_started":1}
I20260928 21:57:52.005934 19708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:52.005709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:42609 (local address 127.2.182.65:36179)
0928 21:57:52.005861 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:52.005894 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:42609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":55,"threads_started":1}
I20260928 21:57:52.818586 19709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:52.818315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:42609 (local address 127.2.182.66:52071)
0928 21:57:52.818490 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:52.818536 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:42609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":72,"threads_started":1}
I20260928 21:57:53.006666 19711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:53.006446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:42609 (local address 127.2.182.65:59529)
0928 21:57:53.006576 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:53.006615 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:42609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":60,"threads_started":1}
I20260928 21:57:53.006666 19710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:53.006439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:42609 (local address 127.2.182.67:55145)
0928 21:57:53.006582 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:53.006618 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:42609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":63,"threads_started":1}
I20260928 21:57:53.819564 19712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:53.819226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:42609 (local address 127.2.182.66:41173)
0928 21:57:53.819469 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:53.819516 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:42609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":69,"threads_started":1}
I20260928 21:57:54.007483 19713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:54.007209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:42609 (local address 127.2.182.67:45619)
0928 21:57:54.007379 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:54.007428 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:42609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":67,"threads_started":1}
I20260928 21:57:54.007616 19714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:54.007209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:42609 (local address 127.2.182.65:58729)
0928 21:57:54.007523 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:54.007565 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:42609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":182,"threads_started":1}
I20260928 21:57:54.820465 19716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:54.820189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:42609 (local address 127.2.182.66:49953)
0928 21:57:54.820358 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:54.820412 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:42609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":82,"threads_started":1}
I20260928 21:57:55.008208 19717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:55.007997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:42609 (local address 127.2.182.67:54359)
0928 21:57:55.008122 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:55.008160 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:42609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":63,"threads_started":1}
I20260928 21:57:55.008265 19718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:55.008064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:42609 (local address 127.2.182.65:44579)
0928 21:57:55.008182 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:55.008222 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:42609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":58,"threads_started":1}
I20260928 21:57:55.821292 19719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:55.821036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:42609 (local address 127.2.182.66:59687)
0928 21:57:55.821198 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:55.821244 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:42609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":67,"threads_started":1}
I20260928 21:57:56.008929 19721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:56.008681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:42609 (local address 127.2.182.65:43381)
0928 21:57:56.008847 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:56.008882 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:42609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":79,"threads_started":1}
I20260928 21:57:56.008929 19720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:56.008681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:42609 (local address 127.2.182.67:42913)
0928 21:57:56.008839 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:56.008886 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:42609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":78,"threads_started":1}
I20260928 21:57:56.822093 19722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:56.821865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:42609 (local address 127.2.182.66:49595)
0928 21:57:56.822005 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:56.822046 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:42609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":69,"threads_started":1}
I20260928 21:57:57.009645 19724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:57.009419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:42609 (local address 127.2.182.65:56751)
0928 21:57:57.009555 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:57.009599 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:42609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":74,"threads_started":1}
I20260928 21:57:57.009647 19723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:57.009405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:42609 (local address 127.2.182.67:48119)
0928 21:57:57.009572 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:57.009607 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:42609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":61,"threads_started":1}
I20260928 21:57:57.822844 19725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:57.822597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:42609 (local address 127.2.182.66:42797)
0928 21:57:57.822748 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:57.822798 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:42609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":67,"threads_started":1}
I20260928 21:57:58.010397 19727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:58.010173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:42609 (local address 127.2.182.67:41735)
0928 21:57:58.010307 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:58.010349 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:42609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":68,"threads_started":1}
I20260928 21:57:58.010401 19726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:58.010173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:42609 (local address 127.2.182.65:40475)
0928 21:57:58.010329 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:58.010360 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:42609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":70,"threads_started":1}
I20260928 21:57:58.823580 19728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:58.823353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:42609 (local address 127.2.182.66:37755)
0928 21:57:58.823493 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:58.823535 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:42609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":68,"threads_started":1}
I20260928 21:57:59.011133 19730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:59.010931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:42609 (local address 127.2.182.65:36729)
0928 21:57:59.011058 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:59.011087 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:42609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":55,"threads_started":1}
I20260928 21:57:59.011138 19729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:57:59.010921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:42609 (local address 127.2.182.67:34523)
0928 21:57:59.011058 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 21:57:59.011093 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.126:42609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":62,"threads_started":1}
I20260928 21:57:59.326956 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 19223
I20260928 21:57:59.331123 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 19391
I20260928 21:57:59.335448 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 19527
2026-09-28T21:57:59Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58186 ms)
[----------] 30 tests from SecurityITest (239288 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:57:59 dist-test-slave-k6lz krb5kdc[19733](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:57:59 dist-test-slave-k6lz krb5kdc[19733](info): set up 2 sockets
Sep 28 21:57:59 dist-test-slave-k6lz krb5kdc[19733](info): commencing operation
krb5kdc: starting...
W20260928 21:58:00.311065 2777 mini_kdc.cc:121] Time spent starting KDC: real 0.954s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Sep 28 21:58:00 dist-test-slave-k6lz krb5kdc[19733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:58:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:58:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab.
Entry for principal kudu/127.2.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab.
I20260928 21:58:00.430959 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:43253
--webserver_interface=127.2.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab
--principal=kudu/127.2.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:35041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.126:43253
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:58:00.515203 19746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:58:00.515472 19746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:58:00.515508 19746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:58:00.516707 19746 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:58:00.516749 19746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:58:00.516768 19746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:58:00.516783 19746 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:58:00.516799 19746 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:58:00.518308 19746 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:35041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.126:43253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab
--principal=kudu/127.2.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:43253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19746
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:58:00.518566 19746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:58:00.518788 19746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:58:00.524160 19752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:00.524165 19751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:00.524160 19754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:00.524178 19746 server_base.cc:1034] running on GCE node
I20260928 21:58:00.524674 19746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:58:00.524935 19746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:58:00.526104 19746 hybrid_clock.cc:648] HybridClock initialized: now 1790632680526086 us; error 39 us; skew 500 ppm
Sep 28 21:58:00 dist-test-slave-k6lz krb5kdc[19733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632680, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:58:00.528736 19746 init.cc:377] Logged in from keytab as kudu/127.2.182.126@KRBTEST.COM (short username kudu)
I20260928 21:58:00.529171 19746 webserver.cc:460] Webserver started at http://127.2.182.126:42603/ using document root <none> and password file <none>
I20260928 21:58:00.529369 19746 fs_manager.cc:359] Metadata directory not provided
I20260928 21:58:00.529419 19746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:58:00.529539 19746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:58:00.530692 19746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/minicluster-data/master-0/data/instance:
uuid: "929112f9f29549969ab27227d7adedc1"
format_stamp: "Formatted at 2026-09-28 21:58:00 on dist-test-slave-k6lz"
server_key: "e17687b4cec72e32662b23f996637c62"
server_key_iv: "7d31327684f0eebb39de983efa72b6ed"
server_key_version: "encryptionkey@0"
I20260928 21:58:00.531038 19746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/minicluster-data/master-0/wal/instance:
uuid: "929112f9f29549969ab27227d7adedc1"
format_stamp: "Formatted at 2026-09-28 21:58:00 on dist-test-slave-k6lz"
server_key: "e17687b4cec72e32662b23f996637c62"
server_key_iv: "7d31327684f0eebb39de983efa72b6ed"
server_key_version: "encryptionkey@0"
I20260928 21:58:00.532455 19746 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 21:58:00.533313 19761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:00.533511 19746 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:58:00.533607 19746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/minicluster-data/master-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/minicluster-data/master-0/wal
uuid: "929112f9f29549969ab27227d7adedc1"
format_stamp: "Formatted at 2026-09-28 21:58:00 on dist-test-slave-k6lz"
server_key: "e17687b4cec72e32662b23f996637c62"
server_key_iv: "7d31327684f0eebb39de983efa72b6ed"
server_key_version: "encryptionkey@0"
I20260928 21:58:00.533684 19746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:58:00.561704 19746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:58:00.564085 19746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:58:00.564230 19746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:58:00.568140 19746 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.126:43253
I20260928 21:58:00.568178 19813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.126:43253 every 8 connection(s)
I20260928 21:58:00.568594 19746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
I20260928 21:58:00.569581 19814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:58:00.572058 19814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 929112f9f29549969ab27227d7adedc1: Bootstrap starting.
I20260928 21:58:00.572602 19814 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 929112f9f29549969ab27227d7adedc1: Neither blocks nor log segments found. Creating new log.
I20260928 21:58:00.572849 19814 log.cc:824] T 00000000000000000000000000000000 P 929112f9f29549969ab27227d7adedc1: Log is configured to *not* fsync() on all Append() calls
I20260928 21:58:00.573565 19814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 929112f9f29549969ab27227d7adedc1: No bootstrap required, opened a new log
I20260928 21:58:00.575294 19814 raft_consensus.cc:348] T 00000000000000000000000000000000 P 929112f9f29549969ab27227d7adedc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "929112f9f29549969ab27227d7adedc1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43253 } }
I20260928 21:58:00.575476 19814 raft_consensus.cc:374] T 00000000000000000000000000000000 P 929112f9f29549969ab27227d7adedc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:58:00.575515 19814 raft_consensus.cc:729] T 00000000000000000000000000000000 P 929112f9f29549969ab27227d7adedc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 929112f9f29549969ab27227d7adedc1, State: Initialized, Role: FOLLOWER
I20260928 21:58:00.575647 19814 consensus_queue.cc:260] T 00000000000000000000000000000000 P 929112f9f29549969ab27227d7adedc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "929112f9f29549969ab27227d7adedc1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43253 } }
I20260928 21:58:00.575707 19814 raft_consensus.cc:388] T 00000000000000000000000000000000 P 929112f9f29549969ab27227d7adedc1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:58:00.575735 19814 raft_consensus.cc:482] T 00000000000000000000000000000000 P 929112f9f29549969ab27227d7adedc1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:58:00.575783 19814 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 929112f9f29549969ab27227d7adedc1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:58:00.576058 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 19746
I20260928 21:58:00.576148 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/minicluster-data/master-0/wal/instance
I20260928 21:58:00.576330 2777 external_mini_cluster.cc:1428] Setting key cb5cad9ee4ed04184c0109d3bc495648
I20260928 21:58:00.576372 19814 raft_consensus.cc:504] T 00000000000000000000000000000000 P 929112f9f29549969ab27227d7adedc1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "929112f9f29549969ab27227d7adedc1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43253 } }
I20260928 21:58:00.576474 19814 leader_election.cc:302] T 00000000000000000000000000000000 P 929112f9f29549969ab27227d7adedc1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 929112f9f29549969ab27227d7adedc1; no voters:
I20260928 21:58:00.576637 19814 leader_election.cc:288] T 00000000000000000000000000000000 P 929112f9f29549969ab27227d7adedc1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:58:00.576675 19817 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 929112f9f29549969ab27227d7adedc1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:58:00.577020 19814 sys_catalog.cc:564] T 00000000000000000000000000000000 P 929112f9f29549969ab27227d7adedc1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:58:00.577056 19817 raft_consensus.cc:686] T 00000000000000000000000000000000 P 929112f9f29549969ab27227d7adedc1 [term 1 LEADER]: Becoming Leader. State: Replica: 929112f9f29549969ab27227d7adedc1, State: Running, Role: LEADER
I20260928 21:58:00.577147 19817 consensus_queue.cc:237] T 00000000000000000000000000000000 P 929112f9f29549969ab27227d7adedc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "929112f9f29549969ab27227d7adedc1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43253 } }
I20260928 21:58:00.577646 19821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 929112f9f29549969ab27227d7adedc1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 929112f9f29549969ab27227d7adedc1. Latest consensus state: current_term: 1 leader_uuid: "929112f9f29549969ab27227d7adedc1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "929112f9f29549969ab27227d7adedc1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43253 } } }
I20260928 21:58:00.577733 19821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 929112f9f29549969ab27227d7adedc1 [sys.catalog]: This master's current role is: LEADER
I20260928 21:58:00.578061 19818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 929112f9f29549969ab27227d7adedc1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "929112f9f29549969ab27227d7adedc1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "929112f9f29549969ab27227d7adedc1" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43253 } } }
I20260928 21:58:00.578140 19818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 929112f9f29549969ab27227d7adedc1 [sys.catalog]: This master's current role is: LEADER
I20260928 21:58:00.578179 19828 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:58:00.578975 19828 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:58:00.580618 19828 catalog_manager.cc:1347] Generated new cluster ID: 29a6a1234f5e470e933fee40c6570cfb
I20260928 21:58:00.580673 19828 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 21:58:00 dist-test-slave-k6lz krb5kdc[19733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:58:00.587689 19820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:00.576808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37520 (local address 127.2.182.126:43253)
0928 21:58:00.578527 (+ 1719us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:00.578535 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:00.578568 (+ 33us) server_negotiation.cc:407] Connection header received
0928 21:58:00.578695 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:00.578703 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:00.578762 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:00.578898 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:00.579575 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:00.580782 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:00.581523 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:00.581956 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:00.582054 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:00.584250 (+ 2196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:00.584256 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:00.584256 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:00.584268 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:00.586228 (+ 1960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:00.586554 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:00.586557 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:00.586558 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:00.586654 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:00.586824 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:00.586824 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:00.586825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:00.587076 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:00.587227 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:00.587319 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 21:58:00.587486 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1676,"thread_start_us":72,"threads_started":1}
I20260928 21:58:00.592084 19828 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:58:00.592636 19828 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:58:00.601784 19828 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 929112f9f29549969ab27227d7adedc1: Generated new TSK 0
I20260928 21:58:00.601989 19828 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.182.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab.
Entry for principal kudu/127.2.182.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab.
I20260928 21:58:00.635499 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.65:0
--local_ip_for_outbound_sockets=127.2.182.65
--webserver_interface=127.2.182.65
--webserver_port=0
--tserver_master_addrs=127.2.182.126:43253
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab
--principal=kudu/127.2.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:35041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:58:00.723470 19840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:58:00.723702 19840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:58:00.723730 19840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:58:00.725063 19840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:58:00.725106 19840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:58:00.725133 19840 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.65
I20260928 21:58:00.726743 19840 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:35041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab
--principal=kudu/127.2.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.182.65
--webserver_port=0
--tserver_master_addrs=127.2.182.126:43253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.65
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:58:00.726979 19840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:58:00.727174 19840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:58:00.732339 19845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:00.732306 19848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:00.732306 19846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:00.732585 19840 server_base.cc:1034] running on GCE node
I20260928 21:58:00.732769 19840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:58:00.733009 19840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:58:00.734180 19840 hybrid_clock.cc:648] HybridClock initialized: now 1790632680734167 us; error 25 us; skew 500 ppm
Sep 28 21:58:00 dist-test-slave-k6lz krb5kdc[19733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632680, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:58:00.736954 19840 init.cc:377] Logged in from keytab as kudu/127.2.182.65@KRBTEST.COM (short username kudu)
I20260928 21:58:00.737478 19840 webserver.cc:460] Webserver started at http://127.2.182.65:35715/ using document root <none> and password file <none>
I20260928 21:58:00.737695 19840 fs_manager.cc:359] Metadata directory not provided
I20260928 21:58:00.737743 19840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:58:00.737865 19840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:58:00.738922 19840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/minicluster-data/ts-0/data/instance:
uuid: "1c23cb7eb2f14f038833761c7a8cc91f"
format_stamp: "Formatted at 2026-09-28 21:58:00 on dist-test-slave-k6lz"
server_key: "a6fc911e15eb3314937433036536d448"
server_key_iv: "05f3c3b6c3eb39c6b5a7f5ad1989c10c"
server_key_version: "encryptionkey@0"
I20260928 21:58:00.739276 19840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/minicluster-data/ts-0/wal/instance:
uuid: "1c23cb7eb2f14f038833761c7a8cc91f"
format_stamp: "Formatted at 2026-09-28 21:58:00 on dist-test-slave-k6lz"
server_key: "a6fc911e15eb3314937433036536d448"
server_key_iv: "05f3c3b6c3eb39c6b5a7f5ad1989c10c"
server_key_version: "encryptionkey@0"
I20260928 21:58:00.740756 19840 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:58:00.741820 19855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:00.742069 19840 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:58:00.742163 19840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/minicluster-data/ts-0/wal
uuid: "1c23cb7eb2f14f038833761c7a8cc91f"
format_stamp: "Formatted at 2026-09-28 21:58:00 on dist-test-slave-k6lz"
server_key: "a6fc911e15eb3314937433036536d448"
server_key_iv: "05f3c3b6c3eb39c6b5a7f5ad1989c10c"
server_key_version: "encryptionkey@0"
I20260928 21:58:00.742233 19840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:58:00.752133 19840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:58:00.754303 19840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:58:00.754680 19840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:58:00.755046 19840 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:58:00.755504 19840 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:58:00.755558 19840 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:00.755591 19840 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:58:00.755616 19840 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:00.761214 19840 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.65:32855
I20260928 21:58:00.761273 19968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.65:32855 every 8 connection(s)
I20260928 21:58:00.761579 19840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
Sep 28 21:58:00 dist-test-slave-k6lz krb5kdc[19733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632680, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.65@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:58:00.771296 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 19840
I20260928 21:58:00.771440 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/minicluster-data/ts-0/wal/instance
I20260928 21:58:00.771672 2777 external_mini_cluster.cc:1428] Setting key 8cd6bb343fc1193eb95e19294f1cfe62
I20260928 21:58:00.772763 19971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:00.762288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:43253 (local address 127.2.182.65:55489)
0928 21:58:00.762461 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:00.762505 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:00.763029 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:00.763376 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:00.763382 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:00.764068 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:58:00.764273 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:00.764279 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:00.765057 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:00.765058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:00.765888 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:00.765890 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:00.766345 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:00.766347 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:00.766527 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:00.767276 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:58:00.767286 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:58:00.768923 (+ 1637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:58:00.770928 (+ 2005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:00.770940 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:00.770942 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:00.771269 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:00.771527 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:00.771528 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:00.771529 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:00.771686 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:00.772039 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:58:00.772048 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:58:00.772190 (+ 142us) client_negotiation.cc:769] Sending connection context
0928 21:58:00.772277 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 21:58:00.772522 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":79,"threads_started":1}
I20260928 21:58:00.772646 19820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:00.762331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.65:55489 (local address 127.2.182.126:43253)
0928 21:58:00.762427 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:00.762429 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:00.762643 (+ 214us) server_negotiation.cc:407] Connection header received
0928 21:58:00.763149 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:00.763151 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:00.763162 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:00.763232 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:00.764414 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:00.764934 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:00.766016 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:00.766225 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:00.766290 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:00.769070 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:00.769072 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:00.769072 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:00.769081 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:00.770811 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:00.771389 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:00.771391 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:00.771391 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:00.771464 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:00.771814 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:00.771815 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:00.771816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:00.771976 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:00.772034 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:00.772336 (+ 302us) server_negotiation.cc:299] Negotiation successful
0928 21:58:00.772432 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 21:58:00.773712 19969 heartbeater.cc:347] Connected to a master server at 127.2.182.126:43253
I20260928 21:58:00.773846 19969 heartbeater.cc:464] Registering TS with master...
I20260928 21:58:00.774070 19969 heartbeater.cc:511] Master 127.2.182.126:43253 requested a full tablet report, sending...
I20260928 21:58:00.774698 19778 ts_manager.cc:194] Registered new tserver with Master: 1c23cb7eb2f14f038833761c7a8cc91f (127.2.182.65:32855)
I20260928 21:58:00.775442 19778 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.65@KRBTEST.COM'} at 127.2.182.65:55489
WARNING: no policy specified for kudu/127.2.182.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab.
Entry for principal kudu/127.2.182.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab.
I20260928 21:58:00.799353 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.66:0
--local_ip_for_outbound_sockets=127.2.182.66
--webserver_interface=127.2.182.66
--webserver_port=0
--tserver_master_addrs=127.2.182.126:43253
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab
--principal=kudu/127.2.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:35041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:58:00.880306 19974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:58:00.880518 19974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:58:00.880549 19974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:58:00.881728 19974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:58:00.881775 19974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:58:00.881811 19974 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.66
I20260928 21:58:00.883263 19974 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:35041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab
--principal=kudu/127.2.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.182.66
--webserver_port=0
--tserver_master_addrs=127.2.182.126:43253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19974
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.66
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:58:00.883519 19974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:58:00.883734 19974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:58:00.888579 19982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:00.888665 19980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:00.888729 19974 server_base.cc:1034] running on GCE node
W20260928 21:58:00.888649 19979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:00.889003 19974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:58:00.889233 19974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:58:00.890372 19974 hybrid_clock.cc:648] HybridClock initialized: now 1790632680890343 us; error 44 us; skew 500 ppm
Sep 28 21:58:00 dist-test-slave-k6lz krb5kdc[19733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632680, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:58:00.892963 19974 init.cc:377] Logged in from keytab as kudu/127.2.182.66@KRBTEST.COM (short username kudu)
I20260928 21:58:00.893394 19974 webserver.cc:460] Webserver started at http://127.2.182.66:34497/ using document root <none> and password file <none>
I20260928 21:58:00.893664 19974 fs_manager.cc:359] Metadata directory not provided
I20260928 21:58:00.893718 19974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:58:00.893836 19974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:58:00.894845 19974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/minicluster-data/ts-1/data/instance:
uuid: "5ab31191c89a454992ad438d689bdf62"
format_stamp: "Formatted at 2026-09-28 21:58:00 on dist-test-slave-k6lz"
server_key: "67b7fe480afa3039dfd47d1516712571"
server_key_iv: "2c31723da9fe8b51e232ce2dae5ae526"
server_key_version: "encryptionkey@0"
I20260928 21:58:00.895160 19974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/minicluster-data/ts-1/wal/instance:
uuid: "5ab31191c89a454992ad438d689bdf62"
format_stamp: "Formatted at 2026-09-28 21:58:00 on dist-test-slave-k6lz"
server_key: "67b7fe480afa3039dfd47d1516712571"
server_key_iv: "2c31723da9fe8b51e232ce2dae5ae526"
server_key_version: "encryptionkey@0"
I20260928 21:58:00.896560 19974 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:58:00.897349 19989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:00.897578 19974 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:58:00.897672 19974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/minicluster-data/ts-1/wal
uuid: "5ab31191c89a454992ad438d689bdf62"
format_stamp: "Formatted at 2026-09-28 21:58:00 on dist-test-slave-k6lz"
server_key: "67b7fe480afa3039dfd47d1516712571"
server_key_iv: "2c31723da9fe8b51e232ce2dae5ae526"
server_key_version: "encryptionkey@0"
I20260928 21:58:00.897739 19974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:58:00.910805 19974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:58:00.912977 19974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:58:00.913115 19974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:58:00.913319 19974 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:58:00.913604 19974 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:58:00.913635 19974 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:00.913663 19974 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:58:00.913681 19974 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:00.919019 19974 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.66:41921
I20260928 21:58:00.919023 20102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.66:41921 every 8 connection(s)
I20260928 21:58:00.919376 19974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
I20260928 21:58:00.924356 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 19974
I20260928 21:58:00.924475 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/minicluster-data/ts-1/wal/instance
I20260928 21:58:00.924713 2777 external_mini_cluster.cc:1428] Setting key 4d9dd46220d01a13f5fe573f3c5b0f5b
Sep 28 21:58:00 dist-test-slave-k6lz krb5kdc[19733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632680, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.66@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:58:00.929145 19820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:00.920008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.66:33871 (local address 127.2.182.126:43253)
0928 21:58:00.920063 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:00.920065 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:00.920281 (+ 216us) server_negotiation.cc:407] Connection header received
0928 21:58:00.920742 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:00.920743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:00.920753 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:00.920821 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:00.921829 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:00.922347 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:00.923717 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:00.923983 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:00.924046 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:00.926441 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:00.926442 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:00.926442 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:00.926455 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:00.927769 (+ 1314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:00.928120 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:00.928121 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:00.928121 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:00.928183 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:00.928368 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:00.928369 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:00.928369 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:00.928495 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:00.928557 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:00.928960 (+ 403us) server_negotiation.cc:299] Negotiation successful
0928 21:58:00.929048 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 21:58:00.929263 20105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:00.919988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:43253 (local address 127.2.182.66:33871)
0928 21:58:00.920136 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:00.920178 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:00.920661 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:00.920879 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:00.920882 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:00.921427 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:58:00.921721 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:00.921726 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:00.922412 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:00.922413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:00.923628 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:00.923632 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:00.924065 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:00.924066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:00.924170 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:00.924661 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:58:00.924670 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:58:00.926331 (+ 1661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:58:00.927836 (+ 1505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:00.927842 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:00.927845 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:00.928055 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:00.928236 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:00.928237 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:00.928237 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:00.928322 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:00.928607 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:58:00.928616 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:58:00.928783 (+ 167us) client_negotiation.cc:769] Sending connection context
0928 21:58:00.928912 (+ 129us) client_negotiation.cc:240] Negotiation successful
0928 21:58:00.929072 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":71,"threads_started":1}
I20260928 21:58:00.929790 20103 heartbeater.cc:347] Connected to a master server at 127.2.182.126:43253
I20260928 21:58:00.929904 20103 heartbeater.cc:464] Registering TS with master...
I20260928 21:58:00.930123 20103 heartbeater.cc:511] Master 127.2.182.126:43253 requested a full tablet report, sending...
I20260928 21:58:00.930560 19778 ts_manager.cc:194] Registered new tserver with Master: 5ab31191c89a454992ad438d689bdf62 (127.2.182.66:41921)
I20260928 21:58:00.931171 19778 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.66@KRBTEST.COM'} at 127.2.182.66:33871
WARNING: no policy specified for kudu/127.2.182.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab.
Entry for principal kudu/127.2.182.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab.
I20260928 21:58:00.951687 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.67:0
--local_ip_for_outbound_sockets=127.2.182.67
--webserver_interface=127.2.182.67
--webserver_port=0
--tserver_master_addrs=127.2.182.126:43253
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab
--principal=kudu/127.2.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:35041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:58:01.032982 20108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:58:01.033226 20108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:58:01.033262 20108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:58:01.034495 20108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:58:01.034548 20108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:58:01.034585 20108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.67
I20260928 21:58:01.036142 20108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:35041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab
--principal=kudu/127.2.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.182.67
--webserver_port=0
--tserver_master_addrs=127.2.182.126:43253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.67
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:58:01.036392 20108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:58:01.036621 20108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:58:01.041813 20113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:01.042014 20116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:01.042147 20108 server_base.cc:1034] running on GCE node
W20260928 21:58:01.042413 20114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:01.042647 20108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:58:01.042932 20108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:58:01.044102 20108 hybrid_clock.cc:648] HybridClock initialized: now 1790632681044090 us; error 31 us; skew 500 ppm
Sep 28 21:58:01 dist-test-slave-k6lz krb5kdc[19733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632681, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:58:01.046825 20108 init.cc:377] Logged in from keytab as kudu/127.2.182.67@KRBTEST.COM (short username kudu)
I20260928 21:58:01.047281 20108 webserver.cc:460] Webserver started at http://127.2.182.67:42619/ using document root <none> and password file <none>
I20260928 21:58:01.047518 20108 fs_manager.cc:359] Metadata directory not provided
I20260928 21:58:01.047564 20108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:58:01.047670 20108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:58:01.048719 20108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/minicluster-data/ts-2/data/instance:
uuid: "f2302da25f4d4ba79e08b01e81f5978d"
format_stamp: "Formatted at 2026-09-28 21:58:01 on dist-test-slave-k6lz"
server_key: "c1ee9688dad41fcc8fdc4562fd186f91"
server_key_iv: "b4e8c1607ca6f45fbe99df2b9d0a8723"
server_key_version: "encryptionkey@0"
I20260928 21:58:01.049045 20108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/minicluster-data/ts-2/wal/instance:
uuid: "f2302da25f4d4ba79e08b01e81f5978d"
format_stamp: "Formatted at 2026-09-28 21:58:01 on dist-test-slave-k6lz"
server_key: "c1ee9688dad41fcc8fdc4562fd186f91"
server_key_iv: "b4e8c1607ca6f45fbe99df2b9d0a8723"
server_key_version: "encryptionkey@0"
I20260928 21:58:01.050462 20108 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:58:01.051854 20123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:01.052115 20108 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:58:01.052217 20108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/minicluster-data/ts-2/wal
uuid: "f2302da25f4d4ba79e08b01e81f5978d"
format_stamp: "Formatted at 2026-09-28 21:58:01 on dist-test-slave-k6lz"
server_key: "c1ee9688dad41fcc8fdc4562fd186f91"
server_key_iv: "b4e8c1607ca6f45fbe99df2b9d0a8723"
server_key_version: "encryptionkey@0"
I20260928 21:58:01.052291 20108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:58:01.070904 20108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:58:01.073975 20108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:58:01.074144 20108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:58:01.074502 20108 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:58:01.074851 20108 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:58:01.074888 20108 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:01.074918 20108 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:58:01.074934 20108 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:01.081436 20108 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.67:33991
I20260928 21:58:01.081568 20236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.67:33991 every 8 connection(s)
I20260928 21:58:01.081825 20108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
I20260928 21:58:01.087158 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 20108
I20260928 21:58:01.087287 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/minicluster-data/ts-2/wal/instance
I20260928 21:58:01.087507 2777 external_mini_cluster.cc:1428] Setting key ebc4bca2f0fe35e6a5f66f48d73245bb
Sep 28 21:58:01 dist-test-slave-k6lz krb5kdc[19733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632681, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.67@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:58:01.091645 19820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:01.082826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.67:45155 (local address 127.2.182.126:43253)
0928 21:58:01.082936 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:01.082939 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:01.083114 (+ 175us) server_negotiation.cc:407] Connection header received
0928 21:58:01.083749 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:01.083750 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:01.083764 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:01.083816 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:01.084867 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:01.085392 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:01.086371 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:01.086566 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:01.086625 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:01.088841 (+ 2216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:01.088842 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:01.088842 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:01.088850 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:01.090167 (+ 1317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:01.090562 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:01.090563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:01.090564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:01.090617 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:01.090840 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:01.090840 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:01.090841 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:01.091048 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:01.091128 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:01.091460 (+ 332us) server_negotiation.cc:299] Negotiation successful
0928 21:58:01.091548 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 21:58:01.091840 20239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:01.082752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:43253 (local address 127.2.182.67:45155)
0928 21:58:01.082981 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:01.083033 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:01.083646 (+ 613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:01.083899 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:01.083902 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:01.084532 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:58:01.084746 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:01.084751 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:01.085488 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:01.085489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:01.086286 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:01.086288 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:01.086640 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:01.086641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:01.086762 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:01.087264 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:58:01.087273 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:58:01.088748 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:58:01.090247 (+ 1499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:01.090255 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:01.090257 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:01.090492 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:01.090671 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:01.090672 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:01.090673 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:01.090783 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:01.091142 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:58:01.091150 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:58:01.091290 (+ 140us) client_negotiation.cc:769] Sending connection context
0928 21:58:01.091418 (+ 128us) client_negotiation.cc:240] Negotiation successful
0928 21:58:01.091619 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":101,"threads_started":1}
I20260928 21:58:01.092446 20237 heartbeater.cc:347] Connected to a master server at 127.2.182.126:43253
I20260928 21:58:01.092646 20237 heartbeater.cc:464] Registering TS with master...
I20260928 21:58:01.092985 20237 heartbeater.cc:511] Master 127.2.182.126:43253 requested a full tablet report, sending...
I20260928 21:58:01.093564 19778 ts_manager.cc:194] Registered new tserver with Master: f2302da25f4d4ba79e08b01e81f5978d (127.2.182.67:33991)
I20260928 21:58:01.093963 19778 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.67@KRBTEST.COM'} at 127.2.182.67:45155
I20260928 21:58:01.099646 2777 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:58:01.108340 19820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:01.101297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37530 (local address 127.2.182.126:43253)
0928 21:58:01.101350 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:01.101352 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:01.101505 (+ 153us) server_negotiation.cc:407] Connection header received
0928 21:58:01.101541 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:01.101542 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:01.101556 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:01.101633 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:01.102330 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:01.102832 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:01.103718 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:01.103912 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:01.103973 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:01.105346 (+ 1373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:01.105347 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:01.105348 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:01.105354 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:01.106818 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:01.107175 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:01.107175 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:01.107176 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:01.107237 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:01.107581 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:01.107582 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:01.107582 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:01.107751 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:01.107833 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:01.108191 (+ 358us) server_negotiation.cc:299] Negotiation successful
0928 21:58:01.108267 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 21:58:01.110245 19778 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37530:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:58:01.112033 19778 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:58:01.118772 20250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:01.114634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46070 (local address 127.2.182.66:41921)
0928 21:58:01.115246 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:01.115250 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:01.115261 (+ 11us) server_negotiation.cc:407] Connection header received
0928 21:58:01.115294 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:01.115296 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:01.115311 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:01.115403 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:58:01.115700 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:01.116480 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:01.117726 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:01.118378 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:01.118447 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:01.118513 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:01.118639 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 21:58:01.118686 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":104,"threads_started":1}
I20260928 21:58:01.118775 20249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:01.114504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.66:41921 (local address 127.0.0.1:46070)
0928 21:58:01.114828 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:01.114866 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:01.114932 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:01.115492 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:01.115499 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:01.115500 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:58:01.115620 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:01.115623 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:01.116573 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:01.116574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:01.117636 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:01.117639 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:01.118456 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:01.118457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:01.118534 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:01.118549 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 21:58:01.118620 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 21:58:01.118678 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":3,"thread_start_us":46,"threads_started":1}
I20260928 21:58:01.119092 20248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:01.114599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.67:33991 (local address 127.0.0.1:47438)
0928 21:58:01.114828 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:01.114861 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:01.114921 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:01.115492 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:01.115495 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:01.115498 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:58:01.115590 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:01.115592 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:01.116876 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:01.116877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:01.117915 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:01.117918 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:01.118854 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:01.118855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:01.118935 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:01.118940 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 21:58:01.118982 (+ 42us) client_negotiation.cc:240] Negotiation successful
0928 21:58:01.119029 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":2,"thread_start_us":76,"threads_started":1}
I20260928 21:58:01.119370 20253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:01.115046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47438 (local address 127.2.182.67:33991)
0928 21:58:01.115222 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:01.115227 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:01.115240 (+ 13us) server_negotiation.cc:407] Connection header received
0928 21:58:01.115280 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:01.115282 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:01.115300 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:01.115405 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:58:01.115660 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:01.116757 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:01.118000 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:01.118711 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:01.118770 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:01.118822 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:01.119236 (+ 414us) server_negotiation.cc:299] Negotiation successful
0928 21:58:01.119279 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":74,"threads_started":1}
I20260928 21:58:01.119647 20037 tablet_service.cc:1467] Processing CreateTablet for tablet ce1db47c19b74b1ca2fb9d91a5bf6398 (DEFAULT_TABLE table=test-table [id=219508c55ed440c9935c53df5222783e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:58:01.120030 20037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce1db47c19b74b1ca2fb9d91a5bf6398. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:58:01.119997 20171 tablet_service.cc:1467] Processing CreateTablet for tablet ce1db47c19b74b1ca2fb9d91a5bf6398 (DEFAULT_TABLE table=test-table [id=219508c55ed440c9935c53df5222783e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:58:01.120291 20171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce1db47c19b74b1ca2fb9d91a5bf6398. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:58:01.123693 20254 tablet_bootstrap.cc:492] T ce1db47c19b74b1ca2fb9d91a5bf6398 P f2302da25f4d4ba79e08b01e81f5978d: Bootstrap starting.
I20260928 21:58:01.124459 20254 tablet_bootstrap.cc:654] T ce1db47c19b74b1ca2fb9d91a5bf6398 P f2302da25f4d4ba79e08b01e81f5978d: Neither blocks nor log segments found. Creating new log.
I20260928 21:58:01.124497 20251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:01.114682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.65:32855 (local address 127.0.0.1:39776)
0928 21:58:01.115516 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:01.115526 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:01.115609 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:01.115809 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:01.115810 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:01.115812 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:58:01.115845 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:01.115846 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:01.117221 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:01.117222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:01.118220 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:01.118223 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:01.124112 (+ 5889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:01.124113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:01.124199 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:01.124205 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 21:58:01.124269 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 21:58:01.124323 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":810,"thread_start_us":61,"threads_started":1}
I20260928 21:58:01.124465 20252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:01.114806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39776 (local address 127.2.182.65:32855)
0928 21:58:01.115000 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:01.115005 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:01.115582 (+ 577us) server_negotiation.cc:407] Connection header received
0928 21:58:01.115712 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:01.115715 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:01.115731 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:01.115803 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:58:01.115952 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:01.117097 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:01.118306 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:01.124021 (+ 5715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:01.124089 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:01.124143 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:01.124340 (+ 197us) server_negotiation.cc:299] Negotiation successful
0928 21:58:01.124394 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":92,"threads_started":1}
I20260928 21:58:01.124970 20254 log.cc:824] T ce1db47c19b74b1ca2fb9d91a5bf6398 P f2302da25f4d4ba79e08b01e81f5978d: Log is configured to *not* fsync() on all Append() calls
I20260928 21:58:01.125175 19903 tablet_service.cc:1467] Processing CreateTablet for tablet ce1db47c19b74b1ca2fb9d91a5bf6398 (DEFAULT_TABLE table=test-table [id=219508c55ed440c9935c53df5222783e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:58:01.125304 20255 tablet_bootstrap.cc:492] T ce1db47c19b74b1ca2fb9d91a5bf6398 P 5ab31191c89a454992ad438d689bdf62: Bootstrap starting.
I20260928 21:58:01.125468 19903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce1db47c19b74b1ca2fb9d91a5bf6398. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:58:01.125726 20254 tablet_bootstrap.cc:492] T ce1db47c19b74b1ca2fb9d91a5bf6398 P f2302da25f4d4ba79e08b01e81f5978d: No bootstrap required, opened a new log
I20260928 21:58:01.125798 20254 ts_tablet_manager.cc:1397] T ce1db47c19b74b1ca2fb9d91a5bf6398 P f2302da25f4d4ba79e08b01e81f5978d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 21:58:01.125988 20255 tablet_bootstrap.cc:654] T ce1db47c19b74b1ca2fb9d91a5bf6398 P 5ab31191c89a454992ad438d689bdf62: Neither blocks nor log segments found. Creating new log.
I20260928 21:58:01.126536 20255 log.cc:824] T ce1db47c19b74b1ca2fb9d91a5bf6398 P 5ab31191c89a454992ad438d689bdf62: Log is configured to *not* fsync() on all Append() calls
I20260928 21:58:01.127482 20255 tablet_bootstrap.cc:492] T ce1db47c19b74b1ca2fb9d91a5bf6398 P 5ab31191c89a454992ad438d689bdf62: No bootstrap required, opened a new log
I20260928 21:58:01.127553 20255 ts_tablet_manager.cc:1397] T ce1db47c19b74b1ca2fb9d91a5bf6398 P 5ab31191c89a454992ad438d689bdf62: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 21:58:01.127863 20254 raft_consensus.cc:348] T ce1db47c19b74b1ca2fb9d91a5bf6398 P f2302da25f4d4ba79e08b01e81f5978d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ab31191c89a454992ad438d689bdf62" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 41921 } } peers { permanent_uuid: "1c23cb7eb2f14f038833761c7a8cc91f" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 32855 } } peers { permanent_uuid: "f2302da25f4d4ba79e08b01e81f5978d" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 33991 } }
I20260928 21:58:01.128039 20254 raft_consensus.cc:374] T ce1db47c19b74b1ca2fb9d91a5bf6398 P f2302da25f4d4ba79e08b01e81f5978d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:58:01.128070 20254 raft_consensus.cc:729] T ce1db47c19b74b1ca2fb9d91a5bf6398 P f2302da25f4d4ba79e08b01e81f5978d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2302da25f4d4ba79e08b01e81f5978d, State: Initialized, Role: FOLLOWER
I20260928 21:58:01.128209 20254 consensus_queue.cc:260] T ce1db47c19b74b1ca2fb9d91a5bf6398 P f2302da25f4d4ba79e08b01e81f5978d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ab31191c89a454992ad438d689bdf62" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 41921 } } peers { permanent_uuid: "1c23cb7eb2f14f038833761c7a8cc91f" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 32855 } } peers { permanent_uuid: "f2302da25f4d4ba79e08b01e81f5978d" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 33991 } }
I20260928 21:58:01.128494 20254 ts_tablet_manager.cc:1428] T ce1db47c19b74b1ca2fb9d91a5bf6398 P f2302da25f4d4ba79e08b01e81f5978d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 21:58:01.128649 20237 heartbeater.cc:503] Master 127.2.182.126:43253 was elected leader, sending a full tablet report...
I20260928 21:58:01.128638 20259 tablet_bootstrap.cc:492] T ce1db47c19b74b1ca2fb9d91a5bf6398 P 1c23cb7eb2f14f038833761c7a8cc91f: Bootstrap starting.
I20260928 21:58:01.129479 20259 tablet_bootstrap.cc:654] T ce1db47c19b74b1ca2fb9d91a5bf6398 P 1c23cb7eb2f14f038833761c7a8cc91f: Neither blocks nor log segments found. Creating new log.
I20260928 21:58:01.129640 20255 raft_consensus.cc:348] T ce1db47c19b74b1ca2fb9d91a5bf6398 P 5ab31191c89a454992ad438d689bdf62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ab31191c89a454992ad438d689bdf62" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 41921 } } peers { permanent_uuid: "1c23cb7eb2f14f038833761c7a8cc91f" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 32855 } } peers { permanent_uuid: "f2302da25f4d4ba79e08b01e81f5978d" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 33991 } }
I20260928 21:58:01.129805 20255 raft_consensus.cc:374] T ce1db47c19b74b1ca2fb9d91a5bf6398 P 5ab31191c89a454992ad438d689bdf62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:58:01.129839 20255 raft_consensus.cc:729] T ce1db47c19b74b1ca2fb9d91a5bf6398 P 5ab31191c89a454992ad438d689bdf62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ab31191c89a454992ad438d689bdf62, State: Initialized, Role: FOLLOWER
I20260928 21:58:01.129956 20255 consensus_queue.cc:260] T ce1db47c19b74b1ca2fb9d91a5bf6398 P 5ab31191c89a454992ad438d689bdf62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ab31191c89a454992ad438d689bdf62" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 41921 } } peers { permanent_uuid: "1c23cb7eb2f14f038833761c7a8cc91f" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 32855 } } peers { permanent_uuid: "f2302da25f4d4ba79e08b01e81f5978d" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 33991 } }
I20260928 21:58:01.130043 20259 log.cc:824] T ce1db47c19b74b1ca2fb9d91a5bf6398 P 1c23cb7eb2f14f038833761c7a8cc91f: Log is configured to *not* fsync() on all Append() calls
I20260928 21:58:01.130223 20255 ts_tablet_manager.cc:1428] T ce1db47c19b74b1ca2fb9d91a5bf6398 P 5ab31191c89a454992ad438d689bdf62: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 21:58:01.130342 20103 heartbeater.cc:503] Master 127.2.182.126:43253 was elected leader, sending a full tablet report...
I20260928 21:58:01.130857 20259 tablet_bootstrap.cc:492] T ce1db47c19b74b1ca2fb9d91a5bf6398 P 1c23cb7eb2f14f038833761c7a8cc91f: No bootstrap required, opened a new log
I20260928 21:58:01.130928 20259 ts_tablet_manager.cc:1397] T ce1db47c19b74b1ca2fb9d91a5bf6398 P 1c23cb7eb2f14f038833761c7a8cc91f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 21:58:01.132334 20259 raft_consensus.cc:348] T ce1db47c19b74b1ca2fb9d91a5bf6398 P 1c23cb7eb2f14f038833761c7a8cc91f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ab31191c89a454992ad438d689bdf62" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 41921 } } peers { permanent_uuid: "1c23cb7eb2f14f038833761c7a8cc91f" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 32855 } } peers { permanent_uuid: "f2302da25f4d4ba79e08b01e81f5978d" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 33991 } }
I20260928 21:58:01.132476 20259 raft_consensus.cc:374] T ce1db47c19b74b1ca2fb9d91a5bf6398 P 1c23cb7eb2f14f038833761c7a8cc91f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:58:01.132498 20259 raft_consensus.cc:729] T ce1db47c19b74b1ca2fb9d91a5bf6398 P 1c23cb7eb2f14f038833761c7a8cc91f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c23cb7eb2f14f038833761c7a8cc91f, State: Initialized, Role: FOLLOWER
I20260928 21:58:01.132609 20259 consensus_queue.cc:260] T ce1db47c19b74b1ca2fb9d91a5bf6398 P 1c23cb7eb2f14f038833761c7a8cc91f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ab31191c89a454992ad438d689bdf62" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 41921 } } peers { permanent_uuid: "1c23cb7eb2f14f038833761c7a8cc91f" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 32855 } } peers { permanent_uuid: "f2302da25f4d4ba79e08b01e81f5978d" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 33991 } }
I20260928 21:58:01.132849 20259 ts_tablet_manager.cc:1428] T ce1db47c19b74b1ca2fb9d91a5bf6398 P 1c23cb7eb2f14f038833761c7a8cc91f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 21:58:01.132952 19969 heartbeater.cc:503] Master 127.2.182.126:43253 was elected leader, sending a full tablet report...
I20260928 21:58:01.138983 20262 raft_consensus.cc:482] T ce1db47c19b74b1ca2fb9d91a5bf6398 P 1c23cb7eb2f14f038833761c7a8cc91f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:58:01.139096 20262 raft_consensus.cc:504] T ce1db47c19b74b1ca2fb9d91a5bf6398 P 1c23cb7eb2f14f038833761c7a8cc91f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ab31191c89a454992ad438d689bdf62" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 41921 } } peers { permanent_uuid: "1c23cb7eb2f14f038833761c7a8cc91f" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 32855 } } peers { permanent_uuid: "f2302da25f4d4ba79e08b01e81f5978d" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 33991 } }
I20260928 21:58:01.139465 20262 leader_election.cc:288] T ce1db47c19b74b1ca2fb9d91a5bf6398 P 1c23cb7eb2f14f038833761c7a8cc91f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ab31191c89a454992ad438d689bdf62 (127.2.182.66:41921), f2302da25f4d4ba79e08b01e81f5978d (127.2.182.67:33991)
I20260928 21:58:01.143412 19971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:01.139618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.67:33991 (local address 127.2.182.65:38707)
0928 21:58:01.139758 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:01.139768 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:01.139826 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:01.140050 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:01.140051 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:01.140054 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:58:01.140133 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:01.140135 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:01.141086 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:01.141086 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:01.142448 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:01.142451 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:01.143126 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:01.143127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:01.143209 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:01.143212 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:58:01.143257 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 21:58:01.143302 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":48}
I20260928 21:58:01.143412 20253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:01.139712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.65:38707 (local address 127.2.182.67:33991)
0928 21:58:01.139766 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:01.139768 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:01.139809 (+ 41us) server_negotiation.cc:407] Connection header received
0928 21:58:01.139957 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:01.139958 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:01.139967 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:01.140024 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:58:01.140190 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:01.140960 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:01.142554 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:01.143054 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:01.143118 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:01.143128 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:01.143271 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 21:58:01.143308 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 21:58:01.143538 20263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:01.139620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.66:41921 (local address 127.2.182.65:44147)
0928 21:58:01.139812 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:01.139821 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:01.139868 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:01.140179 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:01.140180 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:01.140182 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:58:01.140235 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:01.140236 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:01.141061 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:01.141062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:01.142447 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:01.142451 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:01.143223 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:01.143224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:01.143303 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:01.143307 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:58:01.143400 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 21:58:01.143458 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":50,"thread_start_us":122,"threads_started":1}
I20260928 21:58:01.143808 20191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce1db47c19b74b1ca2fb9d91a5bf6398" candidate_uuid: "1c23cb7eb2f14f038833761c7a8cc91f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2302da25f4d4ba79e08b01e81f5978d" is_pre_election: true
I20260928 21:58:01.143934 20250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:01.139671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.65:44147 (local address 127.2.182.66:41921)
0928 21:58:01.139724 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:01.139726 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:01.139860 (+ 134us) server_negotiation.cc:407] Connection header received
0928 21:58:01.140093 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:01.140095 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:01.140103 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:01.140166 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:58:01.140291 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:01.140975 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:01.142586 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:01.143144 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:01.143214 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:01.143223 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:01.143412 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 21:58:01.143454 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 21:58:01.143971 20191 raft_consensus.cc:2454] T ce1db47c19b74b1ca2fb9d91a5bf6398 P f2302da25f4d4ba79e08b01e81f5978d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c23cb7eb2f14f038833761c7a8cc91f in term 0.
I20260928 21:58:01.144196 19856 leader_election.cc:302] T ce1db47c19b74b1ca2fb9d91a5bf6398 P 1c23cb7eb2f14f038833761c7a8cc91f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c23cb7eb2f14f038833761c7a8cc91f, f2302da25f4d4ba79e08b01e81f5978d; no voters:
I20260928 21:58:01.144224 20057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce1db47c19b74b1ca2fb9d91a5bf6398" candidate_uuid: "1c23cb7eb2f14f038833761c7a8cc91f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ab31191c89a454992ad438d689bdf62" is_pre_election: true
I20260928 21:58:01.144351 20057 raft_consensus.cc:2454] T ce1db47c19b74b1ca2fb9d91a5bf6398 P 5ab31191c89a454992ad438d689bdf62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c23cb7eb2f14f038833761c7a8cc91f in term 0.
I20260928 21:58:01.144362 20262 raft_consensus.cc:2781] T ce1db47c19b74b1ca2fb9d91a5bf6398 P 1c23cb7eb2f14f038833761c7a8cc91f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:58:01.144438 20262 raft_consensus.cc:482] T ce1db47c19b74b1ca2fb9d91a5bf6398 P 1c23cb7eb2f14f038833761c7a8cc91f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:58:01.144472 20262 raft_consensus.cc:3037] T ce1db47c19b74b1ca2fb9d91a5bf6398 P 1c23cb7eb2f14f038833761c7a8cc91f [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:58:01.145264 20262 raft_consensus.cc:504] T ce1db47c19b74b1ca2fb9d91a5bf6398 P 1c23cb7eb2f14f038833761c7a8cc91f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ab31191c89a454992ad438d689bdf62" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 41921 } } peers { permanent_uuid: "1c23cb7eb2f14f038833761c7a8cc91f" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 32855 } } peers { permanent_uuid: "f2302da25f4d4ba79e08b01e81f5978d" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 33991 } }
I20260928 21:58:01.145383 20262 leader_election.cc:288] T ce1db47c19b74b1ca2fb9d91a5bf6398 P 1c23cb7eb2f14f038833761c7a8cc91f [CANDIDATE]: Term 1 election: Requested vote from peers 5ab31191c89a454992ad438d689bdf62 (127.2.182.66:41921), f2302da25f4d4ba79e08b01e81f5978d (127.2.182.67:33991)
I20260928 21:58:01.145553 20057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce1db47c19b74b1ca2fb9d91a5bf6398" candidate_uuid: "1c23cb7eb2f14f038833761c7a8cc91f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ab31191c89a454992ad438d689bdf62"
I20260928 21:58:01.145558 20191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce1db47c19b74b1ca2fb9d91a5bf6398" candidate_uuid: "1c23cb7eb2f14f038833761c7a8cc91f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2302da25f4d4ba79e08b01e81f5978d"
I20260928 21:58:01.145627 20191 raft_consensus.cc:3037] T ce1db47c19b74b1ca2fb9d91a5bf6398 P f2302da25f4d4ba79e08b01e81f5978d [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:58:01.145628 20057 raft_consensus.cc:3037] T ce1db47c19b74b1ca2fb9d91a5bf6398 P 5ab31191c89a454992ad438d689bdf62 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:58:01.146369 20191 raft_consensus.cc:2454] T ce1db47c19b74b1ca2fb9d91a5bf6398 P f2302da25f4d4ba79e08b01e81f5978d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c23cb7eb2f14f038833761c7a8cc91f in term 1.
I20260928 21:58:01.146387 20057 raft_consensus.cc:2454] T ce1db47c19b74b1ca2fb9d91a5bf6398 P 5ab31191c89a454992ad438d689bdf62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c23cb7eb2f14f038833761c7a8cc91f in term 1.
I20260928 21:58:01.146528 19856 leader_election.cc:302] T ce1db47c19b74b1ca2fb9d91a5bf6398 P 1c23cb7eb2f14f038833761c7a8cc91f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c23cb7eb2f14f038833761c7a8cc91f, f2302da25f4d4ba79e08b01e81f5978d; no voters:
I20260928 21:58:01.146646 20262 raft_consensus.cc:2781] T ce1db47c19b74b1ca2fb9d91a5bf6398 P 1c23cb7eb2f14f038833761c7a8cc91f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:58:01.146812 20262 raft_consensus.cc:686] T ce1db47c19b74b1ca2fb9d91a5bf6398 P 1c23cb7eb2f14f038833761c7a8cc91f [term 1 LEADER]: Becoming Leader. State: Replica: 1c23cb7eb2f14f038833761c7a8cc91f, State: Running, Role: LEADER
I20260928 21:58:01.146893 20262 consensus_queue.cc:237] T ce1db47c19b74b1ca2fb9d91a5bf6398 P 1c23cb7eb2f14f038833761c7a8cc91f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ab31191c89a454992ad438d689bdf62" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 41921 } } peers { permanent_uuid: "1c23cb7eb2f14f038833761c7a8cc91f" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 32855 } } peers { permanent_uuid: "f2302da25f4d4ba79e08b01e81f5978d" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 33991 } }
I20260928 21:58:01.147771 19778 catalog_manager.cc:5495] T ce1db47c19b74b1ca2fb9d91a5bf6398 P 1c23cb7eb2f14f038833761c7a8cc91f reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c23cb7eb2f14f038833761c7a8cc91f (127.2.182.65). New cstate: current_term: 1 leader_uuid: "1c23cb7eb2f14f038833761c7a8cc91f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ab31191c89a454992ad438d689bdf62" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 41921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c23cb7eb2f14f038833761c7a8cc91f" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 32855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f2302da25f4d4ba79e08b01e81f5978d" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 33991 } health_report { overall_health: UNKNOWN } } }
I20260928 21:58:01.154932 20252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:01.152558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39778 (local address 127.2.182.65:32855)
0928 21:58:01.152613 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:01.152615 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:01.152622 (+ 7us) server_negotiation.cc:407] Connection header received
0928 21:58:01.152651 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:01.152652 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:01.152667 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:01.152724 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:58:01.152834 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:01.153311 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:01.154062 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:01.154270 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:01.154334 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:01.154548 (+ 214us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:58:01.154668 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:58:01.154738 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:01.154780 (+ 42us) server_negotiation.cc:299] Negotiation successful
0928 21:58:01.154801 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 21:58:01.156916 20191 raft_consensus.cc:1261] T ce1db47c19b74b1ca2fb9d91a5bf6398 P f2302da25f4d4ba79e08b01e81f5978d [term 1 FOLLOWER]: Refusing update from remote peer 1c23cb7eb2f14f038833761c7a8cc91f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:58:01.156948 20057 raft_consensus.cc:1261] T ce1db47c19b74b1ca2fb9d91a5bf6398 P 5ab31191c89a454992ad438d689bdf62 [term 1 FOLLOWER]: Refusing update from remote peer 1c23cb7eb2f14f038833761c7a8cc91f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:58:01.157215 20262 consensus_queue.cc:1035] T ce1db47c19b74b1ca2fb9d91a5bf6398 P 1c23cb7eb2f14f038833761c7a8cc91f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ab31191c89a454992ad438d689bdf62" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 41921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:58:01.157361 20264 consensus_queue.cc:1035] T ce1db47c19b74b1ca2fb9d91a5bf6398 P 1c23cb7eb2f14f038833761c7a8cc91f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2302da25f4d4ba79e08b01e81f5978d" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 33991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:58:01.160789 20269 mvcc.cc:205] Tried to move back new op lower bound from 7334431462015975424 to 7334431461978267648. Current Snapshot: MvccSnapshot[applied={T|T < 7334431462015975424 or (T in {7334431462015975424})}]
I20260928 21:58:01.161865 19778 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37530:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:58:01.161971 19778 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37530:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:58:01.162757 19778 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 929112f9f29549969ab27227d7adedc1: Sending DeleteTablet for 3 replicas of tablet ce1db47c19b74b1ca2fb9d91a5bf6398
I20260928 21:58:01.163089 20037 tablet_service.cc:1514] Processing DeleteTablet for tablet ce1db47c19b74b1ca2fb9d91a5bf6398 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:58:01 UTC) from {username='kudu', principal='kudu/127.2.182.126@KRBTEST.COM'} at 127.0.0.1:46070
I20260928 21:58:01.163100 20171 tablet_service.cc:1514] Processing DeleteTablet for tablet ce1db47c19b74b1ca2fb9d91a5bf6398 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:58:01 UTC) from {username='kudu', principal='kudu/127.2.182.126@KRBTEST.COM'} at 127.0.0.1:47438
I20260928 21:58:01.163432 19903 tablet_service.cc:1514] Processing DeleteTablet for tablet ce1db47c19b74b1ca2fb9d91a5bf6398 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:58:01 UTC) from {username='kudu', principal='kudu/127.2.182.126@KRBTEST.COM'} at 127.0.0.1:39776
I20260928 21:58:01.163477 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 19840
I20260928 21:58:01.163565 20278 tablet_replica.cc:330] T ce1db47c19b74b1ca2fb9d91a5bf6398 P f2302da25f4d4ba79e08b01e81f5978d: stopping tablet replica
I20260928 21:58:01.164291 20277 tablet_replica.cc:330] T ce1db47c19b74b1ca2fb9d91a5bf6398 P 5ab31191c89a454992ad438d689bdf62: stopping tablet replica
I20260928 21:58:01.164390 20277 raft_consensus.cc:2229] T ce1db47c19b74b1ca2fb9d91a5bf6398 P 5ab31191c89a454992ad438d689bdf62 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:58:01.164502 20277 raft_consensus.cc:2258] T ce1db47c19b74b1ca2fb9d91a5bf6398 P 5ab31191c89a454992ad438d689bdf62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:58:01.164587 20278 raft_consensus.cc:2229] T ce1db47c19b74b1ca2fb9d91a5bf6398 P f2302da25f4d4ba79e08b01e81f5978d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:58:01.164687 20278 raft_consensus.cc:2258] T ce1db47c19b74b1ca2fb9d91a5bf6398 P f2302da25f4d4ba79e08b01e81f5978d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:58:01.165113 20277 ts_tablet_manager.cc:1905] T ce1db47c19b74b1ca2fb9d91a5bf6398 P 5ab31191c89a454992ad438d689bdf62: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:58:01.165112 20278 ts_tablet_manager.cc:1905] T ce1db47c19b74b1ca2fb9d91a5bf6398 P f2302da25f4d4ba79e08b01e81f5978d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:58:01.167129 20277 ts_tablet_manager.cc:1918] T ce1db47c19b74b1ca2fb9d91a5bf6398 P 5ab31191c89a454992ad438d689bdf62: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:58:01.167191 20278 ts_tablet_manager.cc:1918] T ce1db47c19b74b1ca2fb9d91a5bf6398 P f2302da25f4d4ba79e08b01e81f5978d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:58:01.167227 20277 log.cc:1196] T ce1db47c19b74b1ca2fb9d91a5bf6398 P 5ab31191c89a454992ad438d689bdf62: Deleting WAL directory at /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/minicluster-data/ts-1/wal/wals/ce1db47c19b74b1ca2fb9d91a5bf6398
I20260928 21:58:01.167284 20278 log.cc:1196] T ce1db47c19b74b1ca2fb9d91a5bf6398 P f2302da25f4d4ba79e08b01e81f5978d: Deleting WAL directory at /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632440055392-2777-0/minicluster-data/ts-2/wal/wals/ce1db47c19b74b1ca2fb9d91a5bf6398
I20260928 21:58:01.167600 20278 ts_tablet_manager.cc:1939] T ce1db47c19b74b1ca2fb9d91a5bf6398 P f2302da25f4d4ba79e08b01e81f5978d: Deleting consensus metadata
I20260928 21:58:01.167604 20277 ts_tablet_manager.cc:1939] T ce1db47c19b74b1ca2fb9d91a5bf6398 P 5ab31191c89a454992ad438d689bdf62: Deleting consensus metadata
I20260928 21:58:01.168090 19763 catalog_manager.cc:4841] TS 5ab31191c89a454992ad438d689bdf62 (127.2.182.66:41921): tablet ce1db47c19b74b1ca2fb9d91a5bf6398 (table test-table [id=219508c55ed440c9935c53df5222783e]) successfully deleted
I20260928 21:58:01.168210 19762 catalog_manager.cc:4841] TS f2302da25f4d4ba79e08b01e81f5978d (127.2.182.67:33991): tablet ce1db47c19b74b1ca2fb9d91a5bf6398 (table test-table [id=219508c55ed440c9935c53df5222783e]) successfully deleted
W20260928 21:58:01.169785 19763 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.182.65:32855 (error 108)
I20260928 21:58:01.170327 20251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:01.170193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.65:32855 (local address 127.0.0.1:39782)
0928 21:58:01.170253 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:01.170296 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.65:32855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 21:58:01.170326 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 19974
W20260928 21:58:01.170449 19763 catalog_manager.cc:4584] TS 1c23cb7eb2f14f038833761c7a8cc91f (127.2.182.65:32855): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ce1db47c19b74b1ca2fb9d91a5bf6398: Network error: Client connection negotiation failed: client connection to 127.2.182.65:32855: connect: Connection refused (error 111)
I20260928 21:58:01.174924 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 20108
I20260928 21:58:01.178978 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 19746
2026-09-28T21:58:01Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1841 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-28T21:58:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:58:01Z Disabled control of system clock
I20260928 21:58:01.205893 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632440055392-2777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632440055392-2777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632440055392-2777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:44989
--webserver_interface=127.2.182.126
--webserver_port=0
--builtin_ntp_servers=127.2.182.84:37217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.126:44989
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 21:58:01.289904 20283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:58:01.290125 20283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:58:01.290145 20283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:58:01.291424 20283 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:58:01.291476 20283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:58:01.291487 20283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:58:01.291499 20283 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:58:01.291509 20283 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:58:01.292887 20283 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:37217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632440055392-2777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632440055392-2777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.126:44989
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:44989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20283
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632440055392-2777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:58:01.293102 20283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:58:01.293352 20283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:58:01.298755 20291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:01.298812 20289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:01.299067 20288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:01.298914 20283 server_base.cc:1034] running on GCE node
I20260928 21:58:01.299437 20283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:58:01.299865 20283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:58:01.301059 20283 hybrid_clock.cc:648] HybridClock initialized: now 1790632681301033 us; error 38 us; skew 500 ppm
I20260928 21:58:01.302374 20283 webserver.cc:460] Webserver started at http://127.2.182.126:40089/ using document root <none> and password file <none>
I20260928 21:58:01.302742 20283 fs_manager.cc:359] Metadata directory not provided
I20260928 21:58:01.302807 20283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:58:01.302942 20283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:58:01.304215 20283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632440055392-2777-0/minicluster-data/master-0/data/instance:
uuid: "12f3ea23412344d0ac3ab77776f05cc3"
format_stamp: "Formatted at 2026-09-28 21:58:01 on dist-test-slave-k6lz"
server_key: "dd3adca56d738a071dc25552a5e7c848"
server_key_iv: "9be45d383e3025b7eb643fbe767f7da0"
server_key_version: "encryptionkey@0"
I20260928 21:58:01.304556 20283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632440055392-2777-0/minicluster-data/master-0/wal/instance:
uuid: "12f3ea23412344d0ac3ab77776f05cc3"
format_stamp: "Formatted at 2026-09-28 21:58:01 on dist-test-slave-k6lz"
server_key: "dd3adca56d738a071dc25552a5e7c848"
server_key_iv: "9be45d383e3025b7eb643fbe767f7da0"
server_key_version: "encryptionkey@0"
I20260928 21:58:01.305898 20283 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:58:01.306802 20297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:01.307030 20283 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:58:01.307124 20283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632440055392-2777-0/minicluster-data/master-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632440055392-2777-0/minicluster-data/master-0/wal
uuid: "12f3ea23412344d0ac3ab77776f05cc3"
format_stamp: "Formatted at 2026-09-28 21:58:01 on dist-test-slave-k6lz"
server_key: "dd3adca56d738a071dc25552a5e7c848"
server_key_iv: "9be45d383e3025b7eb643fbe767f7da0"
server_key_version: "encryptionkey@0"
I20260928 21:58:01.307216 20283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632440055392-2777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632440055392-2777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632440055392-2777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:58:01.317013 20283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:58:01.317370 20283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:58:01.317504 20283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:58:01.321300 20283 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.126:44989
I20260928 21:58:01.321341 20349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.126:44989 every 8 connection(s)
I20260928 21:58:01.321652 20283 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
I20260928 21:58:01.322700 20350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:58:01.324965 20350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12f3ea23412344d0ac3ab77776f05cc3: Bootstrap starting.
I20260928 21:58:01.325515 20350 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12f3ea23412344d0ac3ab77776f05cc3: Neither blocks nor log segments found. Creating new log.
I20260928 21:58:01.325778 20350 log.cc:824] T 00000000000000000000000000000000 P 12f3ea23412344d0ac3ab77776f05cc3: Log is configured to *not* fsync() on all Append() calls
I20260928 21:58:01.326462 20350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12f3ea23412344d0ac3ab77776f05cc3: No bootstrap required, opened a new log
I20260928 21:58:01.328214 20350 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12f3ea23412344d0ac3ab77776f05cc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12f3ea23412344d0ac3ab77776f05cc3" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 44989 } }
I20260928 21:58:01.328358 20350 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12f3ea23412344d0ac3ab77776f05cc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:58:01.328379 20350 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12f3ea23412344d0ac3ab77776f05cc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12f3ea23412344d0ac3ab77776f05cc3, State: Initialized, Role: FOLLOWER
I20260928 21:58:01.328516 20350 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12f3ea23412344d0ac3ab77776f05cc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12f3ea23412344d0ac3ab77776f05cc3" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 44989 } }
I20260928 21:58:01.328585 20350 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12f3ea23412344d0ac3ab77776f05cc3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:58:01.328606 20350 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12f3ea23412344d0ac3ab77776f05cc3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:58:01.328650 20350 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12f3ea23412344d0ac3ab77776f05cc3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:58:01.329553 20350 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12f3ea23412344d0ac3ab77776f05cc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12f3ea23412344d0ac3ab77776f05cc3" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 44989 } }
I20260928 21:58:01.329845 20350 leader_election.cc:302] T 00000000000000000000000000000000 P 12f3ea23412344d0ac3ab77776f05cc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12f3ea23412344d0ac3ab77776f05cc3; no voters:
I20260928 21:58:01.330090 20350 leader_election.cc:288] T 00000000000000000000000000000000 P 12f3ea23412344d0ac3ab77776f05cc3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:58:01.330216 20353 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12f3ea23412344d0ac3ab77776f05cc3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:58:01.330322 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 20283
I20260928 21:58:01.330390 20353 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12f3ea23412344d0ac3ab77776f05cc3 [term 1 LEADER]: Becoming Leader. State: Replica: 12f3ea23412344d0ac3ab77776f05cc3, State: Running, Role: LEADER
I20260928 21:58:01.330420 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632440055392-2777-0/minicluster-data/master-0/wal/instance
I20260928 21:58:01.330435 20350 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12f3ea23412344d0ac3ab77776f05cc3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:58:01.330506 20353 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12f3ea23412344d0ac3ab77776f05cc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12f3ea23412344d0ac3ab77776f05cc3" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 44989 } }
I20260928 21:58:01.330621 2777 external_mini_cluster.cc:1428] Setting key f710f68f4759a02d37e87f788fcde262
I20260928 21:58:01.331360 20355 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12f3ea23412344d0ac3ab77776f05cc3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12f3ea23412344d0ac3ab77776f05cc3. Latest consensus state: current_term: 1 leader_uuid: "12f3ea23412344d0ac3ab77776f05cc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12f3ea23412344d0ac3ab77776f05cc3" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 44989 } } }
I20260928 21:58:01.331547 20355 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12f3ea23412344d0ac3ab77776f05cc3 [sys.catalog]: This master's current role is: LEADER
I20260928 21:58:01.331773 20354 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12f3ea23412344d0ac3ab77776f05cc3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12f3ea23412344d0ac3ab77776f05cc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12f3ea23412344d0ac3ab77776f05cc3" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 44989 } } }
I20260928 21:58:01.331871 20354 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12f3ea23412344d0ac3ab77776f05cc3 [sys.catalog]: This master's current role is: LEADER
I20260928 21:58:01.331768 20368 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:58:01.332506 20368 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:58:01.334115 20368 catalog_manager.cc:1347] Generated new cluster ID: 69c41ec0a42f40b7a52c3dd2652bc9ca
I20260928 21:58:01.334168 20368 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:58:01.336299 20366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:01.331109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51352 (local address 127.2.182.126:44989)
0928 21:58:01.332016 (+ 907us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:01.332025 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:01.332062 (+ 37us) server_negotiation.cc:407] Connection header received
0928 21:58:01.332179 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:01.332187 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:01.332256 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:01.332369 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:01.332580 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:01.333699 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:01.334572 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:01.334970 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:01.335075 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:01.335725 (+ 650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:01.335729 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:01.335729 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:58:01.335892 (+ 163us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:01.335925 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:58:01.335936 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:01.335990 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:01.336058 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 21:58:01.336136 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":834,"thread_start_us":509,"threads_started":1}
I20260928 21:58:01.342906 20368 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:58:01.343600 20368 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:58:01.350106 20368 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12f3ea23412344d0ac3ab77776f05cc3: Generated new TSK 0
I20260928 21:58:01.350306 20368 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:58:01.352787 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632440055392-2777-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632440055392-2777-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632440055392-2777-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.65:0
--local_ip_for_outbound_sockets=127.2.182.65
--webserver_interface=127.2.182.65
--webserver_port=0
--tserver_master_addrs=127.2.182.126:44989
--builtin_ntp_servers=127.2.182.84:37217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 21:58:01.448015 20374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:58:01.448225 20374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:58:01.448247 20374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:58:01.449591 20374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:58:01.449640 20374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:58:01.449667 20374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.65
I20260928 21:58:01.451161 20374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:37217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632440055392-2777-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632440055392-2777-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.182.65
--webserver_port=0
--tserver_master_addrs=127.2.182.126:44989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.65
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632440055392-2777-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:58:01.451421 20374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:58:01.451727 20374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:58:01.457911 20380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:01.457907 20382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:01.457916 20379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:01.458132 20374 server_base.cc:1034] running on GCE node
I20260928 21:58:01.458483 20374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:58:01.458786 20374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:58:01.459965 20374 hybrid_clock.cc:648] HybridClock initialized: now 1790632681459949 us; error 37 us; skew 500 ppm
I20260928 21:58:01.461678 20374 webserver.cc:460] Webserver started at http://127.2.182.65:39105/ using document root <none> and password file <none>
I20260928 21:58:01.461967 20374 fs_manager.cc:359] Metadata directory not provided
I20260928 21:58:01.462034 20374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:58:01.462159 20374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:58:01.463652 20374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632440055392-2777-0/minicluster-data/ts-0/data/instance:
uuid: "10e633d072bf4474a97a3bf4655baf6f"
format_stamp: "Formatted at 2026-09-28 21:58:01 on dist-test-slave-k6lz"
server_key: "46e441a834b3800a0a0302254938bc85"
server_key_iv: "aabcdf56a51807d109dbbf331e6b7427"
server_key_version: "encryptionkey@0"
I20260928 21:58:01.464058 20374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632440055392-2777-0/minicluster-data/ts-0/wal/instance:
uuid: "10e633d072bf4474a97a3bf4655baf6f"
format_stamp: "Formatted at 2026-09-28 21:58:01 on dist-test-slave-k6lz"
server_key: "46e441a834b3800a0a0302254938bc85"
server_key_iv: "aabcdf56a51807d109dbbf331e6b7427"
server_key_version: "encryptionkey@0"
I20260928 21:58:01.465735 20374 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:58:01.466645 20388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:01.466838 20374 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:58:01.466925 20374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632440055392-2777-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632440055392-2777-0/minicluster-data/ts-0/wal
uuid: "10e633d072bf4474a97a3bf4655baf6f"
format_stamp: "Formatted at 2026-09-28 21:58:01 on dist-test-slave-k6lz"
server_key: "46e441a834b3800a0a0302254938bc85"
server_key_iv: "aabcdf56a51807d109dbbf331e6b7427"
server_key_version: "encryptionkey@0"
I20260928 21:58:01.466995 20374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632440055392-2777-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632440055392-2777-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632440055392-2777-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:58:01.487457 20374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:58:01.487776 20374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:58:01.487902 20374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:58:01.488117 20374 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:58:01.488463 20374 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:58:01.488500 20374 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:01.488523 20374 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:58:01.488534 20374 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:01.493788 20374 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.65:42539
I20260928 21:58:01.493860 20501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.65:42539 every 8 connection(s)
I20260928 21:58:01.494143 20374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
I20260928 21:58:01.498668 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 20374
I20260928 21:58:01.498755 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632440055392-2777-0/minicluster-data/ts-0/wal/instance
I20260928 21:58:01.498960 2777 external_mini_cluster.cc:1428] Setting key 6cce6b821e99aa202029280f631296af
I20260928 21:58:01.499948 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632440055392-2777-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632440055392-2777-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632440055392-2777-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.66:0
--local_ip_for_outbound_sockets=127.2.182.66
--webserver_interface=127.2.182.66
--webserver_port=0
--tserver_master_addrs=127.2.182.126:44989
--builtin_ntp_servers=127.2.182.84:37217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 21:58:01.500119 20366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:01.494968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.65:35427 (local address 127.2.182.126:44989)
0928 21:58:01.495037 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:01.495040 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:01.495465 (+ 425us) server_negotiation.cc:407] Connection header received
0928 21:58:01.495943 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:01.495944 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:01.495961 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:01.496020 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:01.496695 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:01.497195 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:01.498429 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:01.498893 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:01.498976 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:01.499830 (+ 854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:01.499832 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:01.499833 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:58:01.499838 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:01.499853 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:58:01.499858 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:01.499922 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:01.500000 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 21:58:01.500042 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 21:58:01.500313 20504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:01.494924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:44989 (local address 127.2.182.65:35427)
0928 21:58:01.495283 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:01.495320 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:01.495848 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:01.496092 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:01.496096 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:01.496105 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:58:01.496608 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:01.496612 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:01.497294 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:01.497295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:01.498254 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:01.498257 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:01.498984 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:01.498986 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:01.499126 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:01.499694 (+ 568us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:58:01.499694 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:58:01.499725 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:58:01.499749 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:58:01.499928 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:58:01.499933 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:58:01.499936 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:58:01.499999 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 21:58:01.500105 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":93,"threads_started":1}
I20260928 21:58:01.500717 20502 heartbeater.cc:347] Connected to a master server at 127.2.182.126:44989
I20260928 21:58:01.500801 20502 heartbeater.cc:464] Registering TS with master...
I20260928 21:58:01.500982 20502 heartbeater.cc:511] Master 127.2.182.126:44989 requested a full tablet report, sending...
I20260928 21:58:01.501492 20314 ts_manager.cc:194] Registered new tserver with Master: 10e633d072bf4474a97a3bf4655baf6f (127.2.182.65:42539)
I20260928 21:58:01.502178 20314 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.182.65:35427
W20260928 21:58:01.584714 20505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:58:01.584926 20505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:58:01.584959 20505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:58:01.586309 20505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:58:01.586366 20505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:58:01.586405 20505 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.66
I20260928 21:58:01.588117 20505 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:37217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632440055392-2777-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632440055392-2777-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.182.66
--webserver_port=0
--tserver_master_addrs=127.2.182.126:44989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20505
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.66
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632440055392-2777-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:58:01.588377 20505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:58:01.588675 20505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:58:01.593710 20511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:01.593706 20513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:01.593807 20505 server_base.cc:1034] running on GCE node
W20260928 21:58:01.593706 20510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:01.594156 20505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:58:01.594372 20505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:58:01.595695 20505 hybrid_clock.cc:648] HybridClock initialized: now 1790632681595677 us; error 26 us; skew 500 ppm
I20260928 21:58:01.596980 20505 webserver.cc:460] Webserver started at http://127.2.182.66:38511/ using document root <none> and password file <none>
I20260928 21:58:01.597239 20505 fs_manager.cc:359] Metadata directory not provided
I20260928 21:58:01.597291 20505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:58:01.597409 20505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:58:01.598537 20505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632440055392-2777-0/minicluster-data/ts-1/data/instance:
uuid: "a9d4eec40ed947bc80a80eab30971f64"
format_stamp: "Formatted at 2026-09-28 21:58:01 on dist-test-slave-k6lz"
server_key: "dd4ff41e43ddb43593f3bcb84bfc04ce"
server_key_iv: "08a42e1ff5469606db1096d0aafdcac2"
server_key_version: "encryptionkey@0"
I20260928 21:58:01.598855 20505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632440055392-2777-0/minicluster-data/ts-1/wal/instance:
uuid: "a9d4eec40ed947bc80a80eab30971f64"
format_stamp: "Formatted at 2026-09-28 21:58:01 on dist-test-slave-k6lz"
server_key: "dd4ff41e43ddb43593f3bcb84bfc04ce"
server_key_iv: "08a42e1ff5469606db1096d0aafdcac2"
server_key_version: "encryptionkey@0"
I20260928 21:58:01.600266 20505 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 21:58:01.601146 20519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:01.601322 20505 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:58:01.601410 20505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632440055392-2777-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632440055392-2777-0/minicluster-data/ts-1/wal
uuid: "a9d4eec40ed947bc80a80eab30971f64"
format_stamp: "Formatted at 2026-09-28 21:58:01 on dist-test-slave-k6lz"
server_key: "dd4ff41e43ddb43593f3bcb84bfc04ce"
server_key_iv: "08a42e1ff5469606db1096d0aafdcac2"
server_key_version: "encryptionkey@0"
I20260928 21:58:01.601475 20505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632440055392-2777-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632440055392-2777-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632440055392-2777-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:58:01.610205 20505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:58:01.610483 20505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:58:01.610594 20505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:58:01.610772 20505 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:58:01.611073 20505 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:58:01.611104 20505 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:01.611125 20505 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:58:01.611137 20505 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:01.616671 20505 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.66:39603
I20260928 21:58:01.616740 20632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.66:39603 every 8 connection(s)
I20260928 21:58:01.617034 20505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
I20260928 21:58:01.622643 20366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:01.617758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.66:51123 (local address 127.2.182.126:44989)
0928 21:58:01.617840 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:01.617843 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:01.618063 (+ 220us) server_negotiation.cc:407] Connection header received
0928 21:58:01.618887 (+ 824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:01.618888 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:01.618900 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:01.618939 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:01.619683 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:01.620199 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:01.621300 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:01.621499 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:01.621568 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:01.622425 (+ 857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:01.622426 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:01.622427 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:58:01.622431 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:01.622445 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:58:01.622448 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:01.622492 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:01.622558 (+ 66us) server_negotiation.cc:299] Negotiation successful
0928 21:58:01.622583 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 21:58:01.622813 20635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:01.617739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:44989 (local address 127.2.182.66:51123)
0928 21:58:01.617930 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:01.617977 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:01.618757 (+ 780us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:01.619036 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:01.619041 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:01.619051 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:58:01.619607 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:01.619612 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:01.620313 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:01.620314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:01.621211 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:01.621213 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:01.621610 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:01.621612 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:01.621738 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:01.622295 (+ 557us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:58:01.622296 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:58:01.622324 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:58:01.622347 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:58:01.622497 (+ 150us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:58:01.622500 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:58:01.622503 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:58:01.622547 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 21:58:01.622626 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":83,"threads_started":1}
I20260928 21:58:01.623260 20633 heartbeater.cc:347] Connected to a master server at 127.2.182.126:44989
I20260928 21:58:01.623402 20633 heartbeater.cc:464] Registering TS with master...
I20260928 21:58:01.623588 20633 heartbeater.cc:511] Master 127.2.182.126:44989 requested a full tablet report, sending...
I20260928 21:58:01.623920 20314 ts_manager.cc:194] Registered new tserver with Master: a9d4eec40ed947bc80a80eab30971f64 (127.2.182.66:39603)
I20260928 21:58:01.624279 20314 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.182.66:51123
I20260928 21:58:01.625499 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 20505
I20260928 21:58:01.625592 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632440055392-2777-0/minicluster-data/ts-1/wal/instance
I20260928 21:58:01.625739 2777 external_mini_cluster.cc:1428] Setting key f765de3469f79e1fb9d9969261d62ee4
I20260928 21:58:01.626591 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632440055392-2777-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632440055392-2777-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632440055392-2777-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.67:0
--local_ip_for_outbound_sockets=127.2.182.67
--webserver_interface=127.2.182.67
--webserver_port=0
--tserver_master_addrs=127.2.182.126:44989
--builtin_ntp_servers=127.2.182.84:37217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 21:58:01.707976 20636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:58:01.708194 20636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:58:01.708216 20636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:58:01.709457 20636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:58:01.709512 20636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:58:01.709540 20636 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.67
I20260928 21:58:01.711051 20636 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:37217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632440055392-2777-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632440055392-2777-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.182.67
--webserver_port=0
--tserver_master_addrs=127.2.182.126:44989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20636
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.67
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632440055392-2777-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:58:01.711258 20636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:58:01.711588 20636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:58:01.716463 20641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:01.716512 20642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:01.716463 20644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:01.716887 20636 server_base.cc:1034] running on GCE node
I20260928 21:58:01.717080 20636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:58:01.717293 20636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:58:01.718536 20636 hybrid_clock.cc:648] HybridClock initialized: now 1790632681718520 us; error 24 us; skew 500 ppm
I20260928 21:58:01.719823 20636 webserver.cc:460] Webserver started at http://127.2.182.67:41593/ using document root <none> and password file <none>
I20260928 21:58:01.720070 20636 fs_manager.cc:359] Metadata directory not provided
I20260928 21:58:01.720124 20636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:58:01.720238 20636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:58:01.721379 20636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632440055392-2777-0/minicluster-data/ts-2/data/instance:
uuid: "ecff628ea5be47efbde10b99cae55621"
format_stamp: "Formatted at 2026-09-28 21:58:01 on dist-test-slave-k6lz"
server_key: "a659f42899bb8e10901935ecbf9e2dff"
server_key_iv: "68be7ced921600577ea1979532d174d3"
server_key_version: "encryptionkey@0"
I20260928 21:58:01.721704 20636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632440055392-2777-0/minicluster-data/ts-2/wal/instance:
uuid: "ecff628ea5be47efbde10b99cae55621"
format_stamp: "Formatted at 2026-09-28 21:58:01 on dist-test-slave-k6lz"
server_key: "a659f42899bb8e10901935ecbf9e2dff"
server_key_iv: "68be7ced921600577ea1979532d174d3"
server_key_version: "encryptionkey@0"
I20260928 21:58:01.722929 20636 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 21:58:01.723795 20650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:01.724011 20636 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 21:58:01.724099 20636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632440055392-2777-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632440055392-2777-0/minicluster-data/ts-2/wal
uuid: "ecff628ea5be47efbde10b99cae55621"
format_stamp: "Formatted at 2026-09-28 21:58:01 on dist-test-slave-k6lz"
server_key: "a659f42899bb8e10901935ecbf9e2dff"
server_key_iv: "68be7ced921600577ea1979532d174d3"
server_key_version: "encryptionkey@0"
I20260928 21:58:01.724162 20636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632440055392-2777-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632440055392-2777-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632440055392-2777-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:58:01.741228 20636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:58:01.741580 20636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:58:01.741714 20636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:58:01.741940 20636 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:58:01.742262 20636 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:58:01.742295 20636 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:01.742324 20636 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:58:01.742348 20636 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:01.748014 20636 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.67:43003
I20260928 21:58:01.748063 20763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.67:43003 every 8 connection(s)
I20260928 21:58:01.748384 20636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
I20260928 21:58:01.751695 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 20636
I20260928 21:58:01.751786 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632440055392-2777-0/minicluster-data/ts-2/wal/instance
I20260928 21:58:01.751971 2777 external_mini_cluster.cc:1428] Setting key 8c73de02b391a43aba331fc695b407d5
I20260928 21:58:01.753662 20366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:01.749204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.67:44765 (local address 127.2.182.126:44989)
0928 21:58:01.749273 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:01.749275 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:01.749443 (+ 168us) server_negotiation.cc:407] Connection header received
0928 21:58:01.749971 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:01.749972 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:01.749985 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:01.750033 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:01.750695 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:01.751200 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:01.752349 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:01.752591 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:01.752648 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:01.753435 (+ 787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:01.753436 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:01.753437 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:58:01.753441 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:01.753457 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:58:01.753461 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:01.753528 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:01.753579 (+ 51us) server_negotiation.cc:299] Negotiation successful
0928 21:58:01.753604 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 21:58:01.753863 20766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:01.749092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:44989 (local address 127.2.182.67:44765)
0928 21:58:01.749314 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:01.749360 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:01.749866 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:01.750102 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:01.750106 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:01.750116 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:58:01.750609 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:01.750613 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:01.751291 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:01.751292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:01.752279 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:01.752283 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:01.752685 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:01.752686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:01.752814 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:01.753283 (+ 469us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:58:01.753283 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:58:01.753311 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:58:01.753334 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:58:01.753527 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:58:01.753530 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:58:01.753532 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 21:58:01.753574 (+ 42us) client_negotiation.cc:240] Negotiation successful
0928 21:58:01.753656 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":79,"threads_started":1}
I20260928 21:58:01.754354 20764 heartbeater.cc:347] Connected to a master server at 127.2.182.126:44989
I20260928 21:58:01.754437 20764 heartbeater.cc:464] Registering TS with master...
I20260928 21:58:01.754587 20764 heartbeater.cc:511] Master 127.2.182.126:44989 requested a full tablet report, sending...
I20260928 21:58:01.754892 20314 ts_manager.cc:194] Registered new tserver with Master: ecff628ea5be47efbde10b99cae55621 (127.2.182.67:43003)
I20260928 21:58:01.755259 20314 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.182.67:44765
I20260928 21:58:01.763994 2777 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:58:01.768088 20366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:01.765401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51368 (local address 127.2.182.126:44989)
0928 21:58:01.765451 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:01.765452 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:01.765482 (+ 30us) server_negotiation.cc:407] Connection header received
0928 21:58:01.765547 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:01.765548 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:01.765558 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:01.765594 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:01.765722 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:01.766198 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:01.766981 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:01.767168 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:01.767232 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:01.767888 (+ 656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:01.767889 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:01.767890 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:58:01.767893 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:01.767907 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:58:01.767911 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:01.767949 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:01.768006 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 21:58:01.768028 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 21:58:01.769704 20314 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51368:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:58:01.771369 20314 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:58:01.778131 20776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:01.773947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.66:39603 (local address 127.0.0.1:41332)
0928 21:58:01.774076 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:01.774086 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:01.774133 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:01.774477 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:01.774478 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:01.774481 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:58:01.774533 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:01.774535 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:01.775743 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:01.775745 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:01.777046 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:01.777049 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:01.777775 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:01.777776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:01.777856 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:01.777871 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 21:58:01.777940 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 21:58:01.778002 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":64,"threads_started":1}
I20260928 21:58:01.778178 20775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:01.773791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45794 (local address 127.2.182.67:43003)
0928 21:58:01.774234 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:01.774238 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:01.774246 (+ 8us) server_negotiation.cc:407] Connection header received
0928 21:58:01.774280 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:01.774282 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:01.774295 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:01.774589 (+ 294us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:58:01.774762 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:01.776031 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:01.777106 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:01.777850 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:01.777928 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:01.778022 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:01.778048 (+ 26us) server_negotiation.cc:299] Negotiation successful
0928 21:58:01.778088 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":109,"threads_started":1}
I20260928 21:58:01.778335 20772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:01.773691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.65:42539 (local address 127.0.0.1:57618)
0928 21:58:01.773941 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:01.773986 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:01.774061 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:01.774433 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:01.774438 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:01.774440 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:58:01.774517 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:01.774520 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:01.776081 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:01.776082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:01.777143 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:01.777145 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:01.778089 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:01.778090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:01.778154 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:01.778158 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:58:01.778222 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 21:58:01.778267 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":76,"threads_started":1}
I20260928 21:58:01.778131 20773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:01.773691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.67:43003 (local address 127.0.0.1:45794)
0928 21:58:01.774058 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:01.774067 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:01.774120 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:01.774654 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:01.774655 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:01.774657 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:58:01.774692 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:01.774694 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:01.776111 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:01.776112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:01.777026 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:01.777029 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:01.777925 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:01.777926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:01.777979 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:01.777982 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:58:01.778018 (+ 36us) client_negotiation.cc:240] Negotiation successful
0928 21:58:01.778048 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":9,"thread_start_us":95,"threads_started":1}
I20260928 21:58:01.778565 20777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:01.774108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41332 (local address 127.2.182.66:39603)
0928 21:58:01.774252 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:01.774257 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:01.774265 (+ 8us) server_negotiation.cc:407] Connection header received
0928 21:58:01.774299 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:01.774300 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:01.774323 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:01.774389 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:58:01.774641 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:01.775656 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:01.777153 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:01.777692 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:01.777757 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:01.777859 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:01.778181 (+ 322us) server_negotiation.cc:299] Negotiation successful
0928 21:58:01.778367 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":82,"threads_started":1}
I20260928 21:58:01.778628 20774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:01.773792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57618 (local address 127.2.182.65:42539)
0928 21:58:01.774006 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:01.774010 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:01.774137 (+ 127us) server_negotiation.cc:407] Connection header received
0928 21:58:01.774185 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:01.774187 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:01.774203 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:01.774269 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:58:01.774607 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:01.775966 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:01.777223 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:01.777927 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:01.777983 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:01.778055 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:01.778497 (+ 442us) server_negotiation.cc:299] Negotiation successful
0928 21:58:01.778545 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":105,"threads_started":1}
I20260928 21:58:01.779134 20698 tablet_service.cc:1467] Processing CreateTablet for tablet 20dc6dfc6921479e9ef92521fce005d0 (DEFAULT_TABLE table=test-table [id=abccdcf5013440fe9a2c49e1b03fa85d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:58:01.779235 20436 tablet_service.cc:1467] Processing CreateTablet for tablet 20dc6dfc6921479e9ef92521fce005d0 (DEFAULT_TABLE table=test-table [id=abccdcf5013440fe9a2c49e1b03fa85d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:58:01.779249 20567 tablet_service.cc:1467] Processing CreateTablet for tablet 20dc6dfc6921479e9ef92521fce005d0 (DEFAULT_TABLE table=test-table [id=abccdcf5013440fe9a2c49e1b03fa85d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:58:01.779568 20567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20dc6dfc6921479e9ef92521fce005d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:58:01.779573 20698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20dc6dfc6921479e9ef92521fce005d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:58:01.779600 20436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20dc6dfc6921479e9ef92521fce005d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:58:01.782361 20778 tablet_bootstrap.cc:492] T 20dc6dfc6921479e9ef92521fce005d0 P 10e633d072bf4474a97a3bf4655baf6f: Bootstrap starting.
I20260928 21:58:01.782992 20780 tablet_bootstrap.cc:492] T 20dc6dfc6921479e9ef92521fce005d0 P a9d4eec40ed947bc80a80eab30971f64: Bootstrap starting.
I20260928 21:58:01.783071 20779 tablet_bootstrap.cc:492] T 20dc6dfc6921479e9ef92521fce005d0 P ecff628ea5be47efbde10b99cae55621: Bootstrap starting.
I20260928 21:58:01.783090 20778 tablet_bootstrap.cc:654] T 20dc6dfc6921479e9ef92521fce005d0 P 10e633d072bf4474a97a3bf4655baf6f: Neither blocks nor log segments found. Creating new log.
I20260928 21:58:01.783412 20778 log.cc:824] T 20dc6dfc6921479e9ef92521fce005d0 P 10e633d072bf4474a97a3bf4655baf6f: Log is configured to *not* fsync() on all Append() calls
I20260928 21:58:01.783666 20779 tablet_bootstrap.cc:654] T 20dc6dfc6921479e9ef92521fce005d0 P ecff628ea5be47efbde10b99cae55621: Neither blocks nor log segments found. Creating new log.
I20260928 21:58:01.783667 20780 tablet_bootstrap.cc:654] T 20dc6dfc6921479e9ef92521fce005d0 P a9d4eec40ed947bc80a80eab30971f64: Neither blocks nor log segments found. Creating new log.
I20260928 21:58:01.783969 20779 log.cc:824] T 20dc6dfc6921479e9ef92521fce005d0 P ecff628ea5be47efbde10b99cae55621: Log is configured to *not* fsync() on all Append() calls
I20260928 21:58:01.783969 20780 log.cc:824] T 20dc6dfc6921479e9ef92521fce005d0 P a9d4eec40ed947bc80a80eab30971f64: Log is configured to *not* fsync() on all Append() calls
I20260928 21:58:01.784094 20778 tablet_bootstrap.cc:492] T 20dc6dfc6921479e9ef92521fce005d0 P 10e633d072bf4474a97a3bf4655baf6f: No bootstrap required, opened a new log
I20260928 21:58:01.784161 20778 ts_tablet_manager.cc:1397] T 20dc6dfc6921479e9ef92521fce005d0 P 10e633d072bf4474a97a3bf4655baf6f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:58:01.784582 20779 tablet_bootstrap.cc:492] T 20dc6dfc6921479e9ef92521fce005d0 P ecff628ea5be47efbde10b99cae55621: No bootstrap required, opened a new log
I20260928 21:58:01.784631 20779 ts_tablet_manager.cc:1397] T 20dc6dfc6921479e9ef92521fce005d0 P ecff628ea5be47efbde10b99cae55621: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 21:58:01.784672 20780 tablet_bootstrap.cc:492] T 20dc6dfc6921479e9ef92521fce005d0 P a9d4eec40ed947bc80a80eab30971f64: No bootstrap required, opened a new log
I20260928 21:58:01.784730 20780 ts_tablet_manager.cc:1397] T 20dc6dfc6921479e9ef92521fce005d0 P a9d4eec40ed947bc80a80eab30971f64: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 21:58:01.786513 20779 raft_consensus.cc:348] T 20dc6dfc6921479e9ef92521fce005d0 P ecff628ea5be47efbde10b99cae55621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecff628ea5be47efbde10b99cae55621" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 43003 } } peers { permanent_uuid: "a9d4eec40ed947bc80a80eab30971f64" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 39603 } } peers { permanent_uuid: "10e633d072bf4474a97a3bf4655baf6f" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 42539 } }
I20260928 21:58:01.786664 20779 raft_consensus.cc:374] T 20dc6dfc6921479e9ef92521fce005d0 P ecff628ea5be47efbde10b99cae55621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:58:01.786695 20779 raft_consensus.cc:729] T 20dc6dfc6921479e9ef92521fce005d0 P ecff628ea5be47efbde10b99cae55621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecff628ea5be47efbde10b99cae55621, State: Initialized, Role: FOLLOWER
I20260928 21:58:01.786711 20778 raft_consensus.cc:348] T 20dc6dfc6921479e9ef92521fce005d0 P 10e633d072bf4474a97a3bf4655baf6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecff628ea5be47efbde10b99cae55621" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 43003 } } peers { permanent_uuid: "a9d4eec40ed947bc80a80eab30971f64" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 39603 } } peers { permanent_uuid: "10e633d072bf4474a97a3bf4655baf6f" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 42539 } }
I20260928 21:58:01.786808 20779 consensus_queue.cc:260] T 20dc6dfc6921479e9ef92521fce005d0 P ecff628ea5be47efbde10b99cae55621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecff628ea5be47efbde10b99cae55621" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 43003 } } peers { permanent_uuid: "a9d4eec40ed947bc80a80eab30971f64" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 39603 } } peers { permanent_uuid: "10e633d072bf4474a97a3bf4655baf6f" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 42539 } }
I20260928 21:58:01.786877 20778 raft_consensus.cc:374] T 20dc6dfc6921479e9ef92521fce005d0 P 10e633d072bf4474a97a3bf4655baf6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:58:01.786911 20778 raft_consensus.cc:729] T 20dc6dfc6921479e9ef92521fce005d0 P 10e633d072bf4474a97a3bf4655baf6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10e633d072bf4474a97a3bf4655baf6f, State: Initialized, Role: FOLLOWER
I20260928 21:58:01.787057 20779 ts_tablet_manager.cc:1428] T 20dc6dfc6921479e9ef92521fce005d0 P ecff628ea5be47efbde10b99cae55621: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260928 21:58:01.787065 20778 consensus_queue.cc:260] T 20dc6dfc6921479e9ef92521fce005d0 P 10e633d072bf4474a97a3bf4655baf6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecff628ea5be47efbde10b99cae55621" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 43003 } } peers { permanent_uuid: "a9d4eec40ed947bc80a80eab30971f64" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 39603 } } peers { permanent_uuid: "10e633d072bf4474a97a3bf4655baf6f" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 42539 } }
I20260928 21:58:01.787127 20780 raft_consensus.cc:348] T 20dc6dfc6921479e9ef92521fce005d0 P a9d4eec40ed947bc80a80eab30971f64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecff628ea5be47efbde10b99cae55621" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 43003 } } peers { permanent_uuid: "a9d4eec40ed947bc80a80eab30971f64" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 39603 } } peers { permanent_uuid: "10e633d072bf4474a97a3bf4655baf6f" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 42539 } }
I20260928 21:58:01.787266 20780 raft_consensus.cc:374] T 20dc6dfc6921479e9ef92521fce005d0 P a9d4eec40ed947bc80a80eab30971f64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:58:01.787272 20764 heartbeater.cc:503] Master 127.2.182.126:44989 was elected leader, sending a full tablet report...
I20260928 21:58:01.787299 20780 raft_consensus.cc:729] T 20dc6dfc6921479e9ef92521fce005d0 P a9d4eec40ed947bc80a80eab30971f64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9d4eec40ed947bc80a80eab30971f64, State: Initialized, Role: FOLLOWER
I20260928 21:58:01.787324 20778 ts_tablet_manager.cc:1428] T 20dc6dfc6921479e9ef92521fce005d0 P 10e633d072bf4474a97a3bf4655baf6f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 21:58:01.787431 20780 consensus_queue.cc:260] T 20dc6dfc6921479e9ef92521fce005d0 P a9d4eec40ed947bc80a80eab30971f64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecff628ea5be47efbde10b99cae55621" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 43003 } } peers { permanent_uuid: "a9d4eec40ed947bc80a80eab30971f64" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 39603 } } peers { permanent_uuid: "10e633d072bf4474a97a3bf4655baf6f" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 42539 } }
I20260928 21:58:01.787472 20502 heartbeater.cc:503] Master 127.2.182.126:44989 was elected leader, sending a full tablet report...
I20260928 21:58:01.787662 20780 ts_tablet_manager.cc:1428] T 20dc6dfc6921479e9ef92521fce005d0 P a9d4eec40ed947bc80a80eab30971f64: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 21:58:01.787804 20633 heartbeater.cc:503] Master 127.2.182.126:44989 was elected leader, sending a full tablet report...
W20260928 21:58:01.867594 20634 tablet.cc:2369] T 20dc6dfc6921479e9ef92521fce005d0 P a9d4eec40ed947bc80a80eab30971f64: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:58:01.868000 20785 raft_consensus.cc:482] T 20dc6dfc6921479e9ef92521fce005d0 P 10e633d072bf4474a97a3bf4655baf6f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:58:01.868103 20785 raft_consensus.cc:504] T 20dc6dfc6921479e9ef92521fce005d0 P 10e633d072bf4474a97a3bf4655baf6f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecff628ea5be47efbde10b99cae55621" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 43003 } } peers { permanent_uuid: "a9d4eec40ed947bc80a80eab30971f64" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 39603 } } peers { permanent_uuid: "10e633d072bf4474a97a3bf4655baf6f" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 42539 } }
I20260928 21:58:01.868463 20785 leader_election.cc:288] T 20dc6dfc6921479e9ef92521fce005d0 P 10e633d072bf4474a97a3bf4655baf6f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecff628ea5be47efbde10b99cae55621 (127.2.182.67:43003), a9d4eec40ed947bc80a80eab30971f64 (127.2.182.66:39603)
I20260928 21:58:01.872112 20504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:01.868745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.67:43003 (local address 127.2.182.65:56041)
0928 21:58:01.868857 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:01.868871 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:01.868976 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:01.869153 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:01.869153 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:01.869156 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:58:01.869256 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:01.869258 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:01.869914 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:01.869915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:01.871075 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:01.871077 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:01.871817 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:01.871818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:01.871884 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:01.871888 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:58:01.872003 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 21:58:01.872048 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260928 21:58:01.872174 20775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:01.868837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.65:56041 (local address 127.2.182.67:43003)
0928 21:58:01.868924 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:01.868926 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:01.868938 (+ 12us) server_negotiation.cc:407] Connection header received
0928 21:58:01.869062 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:01.869063 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:01.869074 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:01.869133 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:58:01.869320 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:01.869839 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:01.871178 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:01.871716 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:01.871795 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:01.871805 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:01.872047 (+ 242us) server_negotiation.cc:299] Negotiation successful
0928 21:58:01.872096 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 21:58:01.872545 20777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:01.868974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.65:47445 (local address 127.2.182.66:39603)
0928 21:58:01.869015 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:01.869017 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:01.869230 (+ 213us) server_negotiation.cc:407] Connection header received
0928 21:58:01.869283 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:01.869284 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:01.869292 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:01.869340 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:58:01.869598 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:01.870236 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:01.871488 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:01.872097 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:01.872171 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:01.872179 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:01.872432 (+ 253us) server_negotiation.cc:299] Negotiation successful
0928 21:58:01.872479 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 21:58:01.872608 20787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:01.868906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.66:39603 (local address 127.2.182.65:47445)
0928 21:58:01.869160 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:01.869169 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:01.869225 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:01.869450 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:01.869451 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:01.869453 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:58:01.869530 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:01.869532 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:01.870391 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:01.870392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:01.871410 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:01.871412 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:01.872211 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:01.872212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:01.872330 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:01.872341 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 21:58:01.872416 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 21:58:01.872530 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":190,"threads_started":1}
I20260928 21:58:01.872581 20718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20dc6dfc6921479e9ef92521fce005d0" candidate_uuid: "10e633d072bf4474a97a3bf4655baf6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecff628ea5be47efbde10b99cae55621" is_pre_election: true
I20260928 21:58:01.872742 20718 raft_consensus.cc:2454] T 20dc6dfc6921479e9ef92521fce005d0 P ecff628ea5be47efbde10b99cae55621 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10e633d072bf4474a97a3bf4655baf6f in term 0.
I20260928 21:58:01.872975 20390 leader_election.cc:302] T 20dc6dfc6921479e9ef92521fce005d0 P 10e633d072bf4474a97a3bf4655baf6f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10e633d072bf4474a97a3bf4655baf6f, ecff628ea5be47efbde10b99cae55621; no voters:
I20260928 21:58:01.873040 20587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20dc6dfc6921479e9ef92521fce005d0" candidate_uuid: "10e633d072bf4474a97a3bf4655baf6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9d4eec40ed947bc80a80eab30971f64" is_pre_election: true
I20260928 21:58:01.873100 20785 raft_consensus.cc:2781] T 20dc6dfc6921479e9ef92521fce005d0 P 10e633d072bf4474a97a3bf4655baf6f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:58:01.873148 20785 raft_consensus.cc:482] T 20dc6dfc6921479e9ef92521fce005d0 P 10e633d072bf4474a97a3bf4655baf6f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:58:01.873178 20785 raft_consensus.cc:3037] T 20dc6dfc6921479e9ef92521fce005d0 P 10e633d072bf4474a97a3bf4655baf6f [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:58:01.873194 20587 raft_consensus.cc:2454] T 20dc6dfc6921479e9ef92521fce005d0 P a9d4eec40ed947bc80a80eab30971f64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10e633d072bf4474a97a3bf4655baf6f in term 0.
I20260928 21:58:01.874045 20785 raft_consensus.cc:504] T 20dc6dfc6921479e9ef92521fce005d0 P 10e633d072bf4474a97a3bf4655baf6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecff628ea5be47efbde10b99cae55621" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 43003 } } peers { permanent_uuid: "a9d4eec40ed947bc80a80eab30971f64" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 39603 } } peers { permanent_uuid: "10e633d072bf4474a97a3bf4655baf6f" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 42539 } }
I20260928 21:58:01.874161 20785 leader_election.cc:288] T 20dc6dfc6921479e9ef92521fce005d0 P 10e633d072bf4474a97a3bf4655baf6f [CANDIDATE]: Term 1 election: Requested vote from peers ecff628ea5be47efbde10b99cae55621 (127.2.182.67:43003), a9d4eec40ed947bc80a80eab30971f64 (127.2.182.66:39603)
I20260928 21:58:01.874351 20718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20dc6dfc6921479e9ef92521fce005d0" candidate_uuid: "10e633d072bf4474a97a3bf4655baf6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecff628ea5be47efbde10b99cae55621"
I20260928 21:58:01.874384 20587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20dc6dfc6921479e9ef92521fce005d0" candidate_uuid: "10e633d072bf4474a97a3bf4655baf6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9d4eec40ed947bc80a80eab30971f64"
I20260928 21:58:01.874428 20718 raft_consensus.cc:3037] T 20dc6dfc6921479e9ef92521fce005d0 P ecff628ea5be47efbde10b99cae55621 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:58:01.874467 20587 raft_consensus.cc:3037] T 20dc6dfc6921479e9ef92521fce005d0 P a9d4eec40ed947bc80a80eab30971f64 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:58:01.875128 20718 raft_consensus.cc:2454] T 20dc6dfc6921479e9ef92521fce005d0 P ecff628ea5be47efbde10b99cae55621 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10e633d072bf4474a97a3bf4655baf6f in term 1.
I20260928 21:58:01.875250 20587 raft_consensus.cc:2454] T 20dc6dfc6921479e9ef92521fce005d0 P a9d4eec40ed947bc80a80eab30971f64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10e633d072bf4474a97a3bf4655baf6f in term 1.
I20260928 21:58:01.875355 20390 leader_election.cc:302] T 20dc6dfc6921479e9ef92521fce005d0 P 10e633d072bf4474a97a3bf4655baf6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10e633d072bf4474a97a3bf4655baf6f, ecff628ea5be47efbde10b99cae55621; no voters:
I20260928 21:58:01.875444 20785 raft_consensus.cc:2781] T 20dc6dfc6921479e9ef92521fce005d0 P 10e633d072bf4474a97a3bf4655baf6f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:58:01.875607 20785 raft_consensus.cc:686] T 20dc6dfc6921479e9ef92521fce005d0 P 10e633d072bf4474a97a3bf4655baf6f [term 1 LEADER]: Becoming Leader. State: Replica: 10e633d072bf4474a97a3bf4655baf6f, State: Running, Role: LEADER
I20260928 21:58:01.875698 20785 consensus_queue.cc:237] T 20dc6dfc6921479e9ef92521fce005d0 P 10e633d072bf4474a97a3bf4655baf6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecff628ea5be47efbde10b99cae55621" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 43003 } } peers { permanent_uuid: "a9d4eec40ed947bc80a80eab30971f64" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 39603 } } peers { permanent_uuid: "10e633d072bf4474a97a3bf4655baf6f" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 42539 } }
I20260928 21:58:01.876659 20313 catalog_manager.cc:5495] T 20dc6dfc6921479e9ef92521fce005d0 P 10e633d072bf4474a97a3bf4655baf6f reported cstate change: term changed from 0 to 1, leader changed from <none> to 10e633d072bf4474a97a3bf4655baf6f (127.2.182.65). New cstate: current_term: 1 leader_uuid: "10e633d072bf4474a97a3bf4655baf6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecff628ea5be47efbde10b99cae55621" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 43003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9d4eec40ed947bc80a80eab30971f64" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 39603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10e633d072bf4474a97a3bf4655baf6f" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 42539 } health_report { overall_health: HEALTHY } } }
I20260928 21:58:01.898866 20774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:01.896266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57622 (local address 127.2.182.65:42539)
0928 21:58:01.896315 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:01.896317 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:01.896324 (+ 7us) server_negotiation.cc:407] Connection header received
0928 21:58:01.896360 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:01.896360 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:01.896379 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:01.896446 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:58:01.896625 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:01.897132 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:01.897925 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:01.898141 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:01.898196 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:01.898411 (+ 215us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:58:01.898529 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:58:01.898673 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:01.898757 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 21:58:01.898781 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 21:58:01.900885 20718 raft_consensus.cc:1261] T 20dc6dfc6921479e9ef92521fce005d0 P ecff628ea5be47efbde10b99cae55621 [term 1 FOLLOWER]: Refusing update from remote peer 10e633d072bf4474a97a3bf4655baf6f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:58:01.900883 20587 raft_consensus.cc:1261] T 20dc6dfc6921479e9ef92521fce005d0 P a9d4eec40ed947bc80a80eab30971f64 [term 1 FOLLOWER]: Refusing update from remote peer 10e633d072bf4474a97a3bf4655baf6f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:58:01.901227 20785 consensus_queue.cc:1035] T 20dc6dfc6921479e9ef92521fce005d0 P 10e633d072bf4474a97a3bf4655baf6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecff628ea5be47efbde10b99cae55621" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 43003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:58:01.901386 20788 consensus_queue.cc:1035] T 20dc6dfc6921479e9ef92521fce005d0 P 10e633d072bf4474a97a3bf4655baf6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9d4eec40ed947bc80a80eab30971f64" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 39603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:58:01.904037 20791 mvcc.cc:205] Tried to move back new op lower bound from 7334431465063104512 to 7334431464963461120. Current Snapshot: MvccSnapshot[applied={T|T < 7334431465063104512}]
I20260928 21:58:01.904852 20792 mvcc.cc:205] Tried to move back new op lower bound from 7334431465063104512 to 7334431464963461120. Current Snapshot: MvccSnapshot[applied={T|T < 7334431465063104512}]
I20260928 21:58:01.906531 20313 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:58:01.906620 20313 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:58:01.907423 20313 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 12f3ea23412344d0ac3ab77776f05cc3: Sending DeleteTablet for 3 replicas of tablet 20dc6dfc6921479e9ef92521fce005d0
I20260928 21:58:01.907836 20436 tablet_service.cc:1514] Processing DeleteTablet for tablet 20dc6dfc6921479e9ef92521fce005d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:58:01 UTC) from {username='slave'} at 127.0.0.1:57618
I20260928 21:58:01.907886 20698 tablet_service.cc:1514] Processing DeleteTablet for tablet 20dc6dfc6921479e9ef92521fce005d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:58:01 UTC) from {username='slave'} at 127.0.0.1:45794
I20260928 21:58:01.908110 20567 tablet_service.cc:1514] Processing DeleteTablet for tablet 20dc6dfc6921479e9ef92521fce005d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:58:01 UTC) from {username='slave'} at 127.0.0.1:41332
I20260928 21:58:01.908326 20803 tablet_replica.cc:330] T 20dc6dfc6921479e9ef92521fce005d0 P a9d4eec40ed947bc80a80eab30971f64: stopping tablet replica
I20260928 21:58:01.908434 20803 raft_consensus.cc:2229] T 20dc6dfc6921479e9ef92521fce005d0 P a9d4eec40ed947bc80a80eab30971f64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:58:01.908569 20803 raft_consensus.cc:2258] T 20dc6dfc6921479e9ef92521fce005d0 P a9d4eec40ed947bc80a80eab30971f64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:58:01.908569 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 20374
I20260928 21:58:01.909426 20803 ts_tablet_manager.cc:1905] T 20dc6dfc6921479e9ef92521fce005d0 P a9d4eec40ed947bc80a80eab30971f64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:58:01.911196 20803 ts_tablet_manager.cc:1918] T 20dc6dfc6921479e9ef92521fce005d0 P a9d4eec40ed947bc80a80eab30971f64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:58:01.911259 20803 log.cc:1196] T 20dc6dfc6921479e9ef92521fce005d0 P a9d4eec40ed947bc80a80eab30971f64: Deleting WAL directory at /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632440055392-2777-0/minicluster-data/ts-1/wal/wals/20dc6dfc6921479e9ef92521fce005d0
I20260928 21:58:01.911489 20803 ts_tablet_manager.cc:1939] T 20dc6dfc6921479e9ef92521fce005d0 P a9d4eec40ed947bc80a80eab30971f64: Deleting consensus metadata
I20260928 21:58:01.911899 20299 catalog_manager.cc:4841] TS a9d4eec40ed947bc80a80eab30971f64 (127.2.182.66:39603): tablet 20dc6dfc6921479e9ef92521fce005d0 (table test-table [id=abccdcf5013440fe9a2c49e1b03fa85d]) successfully deleted
W20260928 21:58:01.913458 20300 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.182.65:42539 (error 108)
I20260928 21:58:01.913650 20802 tablet_replica.cc:330] T 20dc6dfc6921479e9ef92521fce005d0 P ecff628ea5be47efbde10b99cae55621: stopping tablet replica
I20260928 21:58:01.913761 20802 raft_consensus.cc:2229] T 20dc6dfc6921479e9ef92521fce005d0 P ecff628ea5be47efbde10b99cae55621 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:58:01.913878 20802 raft_consensus.cc:2258] T 20dc6dfc6921479e9ef92521fce005d0 P ecff628ea5be47efbde10b99cae55621 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:58:01.914059 20773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:01.913920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.65:42539 (local address 127.0.0.1:57634)
0928 21:58:01.913965 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:01.914014 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.65:42539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260928 21:58:01.914055 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 20505
W20260928 21:58:01.914161 20300 catalog_manager.cc:4584] TS 10e633d072bf4474a97a3bf4655baf6f (127.2.182.65:42539): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 20dc6dfc6921479e9ef92521fce005d0: Network error: Client connection negotiation failed: client connection to 127.2.182.65:42539: connect: Connection refused (error 111)
I20260928 21:58:01.914825 20802 ts_tablet_manager.cc:1905] T 20dc6dfc6921479e9ef92521fce005d0 P ecff628ea5be47efbde10b99cae55621: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:58:01.916643 20802 ts_tablet_manager.cc:1918] T 20dc6dfc6921479e9ef92521fce005d0 P ecff628ea5be47efbde10b99cae55621: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:58:01.916724 20802 log.cc:1196] T 20dc6dfc6921479e9ef92521fce005d0 P ecff628ea5be47efbde10b99cae55621: Deleting WAL directory at /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632440055392-2777-0/minicluster-data/ts-2/wal/wals/20dc6dfc6921479e9ef92521fce005d0
I20260928 21:58:01.916927 20802 ts_tablet_manager.cc:1939] T 20dc6dfc6921479e9ef92521fce005d0 P ecff628ea5be47efbde10b99cae55621: Deleting consensus metadata
I20260928 21:58:01.917388 20299 catalog_manager.cc:4841] TS ecff628ea5be47efbde10b99cae55621 (127.2.182.67:43003): tablet 20dc6dfc6921479e9ef92521fce005d0 (table test-table [id=abccdcf5013440fe9a2c49e1b03fa85d]) successfully deleted
I20260928 21:58:01.918941 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 20636
I20260928 21:58:01.923226 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 20283
2026-09-28T21:58:01Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (742 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:58:01 dist-test-slave-k6lz krb5kdc[20807](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:58:01 dist-test-slave-k6lz krb5kdc[20807](info): set up 2 sockets
Sep 28 21:58:01 dist-test-slave-k6lz krb5kdc[20807](info): commencing operation
krb5kdc: starting...
W20260928 21:58:02.883003 2777 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Sep 28 21:58:02 dist-test-slave-k6lz krb5kdc[20807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:58:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:58:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab.
Entry for principal kudu/127.2.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab.
I20260928 21:58:03.003604 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:45007
--webserver_interface=127.2.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab
--principal=kudu/127.2.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:43241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.126:45007
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:58:03.084270 20820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:58:03.084497 20820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:58:03.084520 20820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:58:03.085678 20820 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:58:03.085716 20820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:58:03.085729 20820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:58:03.085741 20820 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:58:03.085752 20820 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:58:03.087224 20820 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:43241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.126:45007
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab
--principal=kudu/127.2.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:45007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20820
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:58:03.087495 20820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:58:03.087723 20820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:58:03.092569 20825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:03.092669 20826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:03.092703 20828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:03.092939 20820 server_base.cc:1034] running on GCE node
I20260928 21:58:03.093152 20820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:58:03.093429 20820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:58:03.094579 20820 hybrid_clock.cc:648] HybridClock initialized: now 1790632683094551 us; error 41 us; skew 500 ppm
Sep 28 21:58:03 dist-test-slave-k6lz krb5kdc[20807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632683, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:58:03.097167 20820 init.cc:377] Logged in from keytab as kudu/127.2.182.126@KRBTEST.COM (short username kudu)
I20260928 21:58:03.097601 20820 webserver.cc:460] Webserver started at http://127.2.182.126:39555/ using document root <none> and password file <none>
I20260928 21:58:03.097826 20820 fs_manager.cc:359] Metadata directory not provided
I20260928 21:58:03.097873 20820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:58:03.098004 20820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:58:03.099054 20820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/minicluster-data/master-0/data/instance:
uuid: "20e375dff94b4f598f1d4bdce221ca0b"
format_stamp: "Formatted at 2026-09-28 21:58:03 on dist-test-slave-k6lz"
server_key: "a0c1822063e91f46500fa079230f6058"
server_key_iv: "a0d5aab9bc818a67e28e61b05860f0cc"
server_key_version: "encryptionkey@0"
I20260928 21:58:03.099416 20820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/minicluster-data/master-0/wal/instance:
uuid: "20e375dff94b4f598f1d4bdce221ca0b"
format_stamp: "Formatted at 2026-09-28 21:58:03 on dist-test-slave-k6lz"
server_key: "a0c1822063e91f46500fa079230f6058"
server_key_iv: "a0d5aab9bc818a67e28e61b05860f0cc"
server_key_version: "encryptionkey@0"
I20260928 21:58:03.100745 20820 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:58:03.101629 20835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:03.101804 20820 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:58:03.101891 20820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/minicluster-data/master-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/minicluster-data/master-0/wal
uuid: "20e375dff94b4f598f1d4bdce221ca0b"
format_stamp: "Formatted at 2026-09-28 21:58:03 on dist-test-slave-k6lz"
server_key: "a0c1822063e91f46500fa079230f6058"
server_key_iv: "a0d5aab9bc818a67e28e61b05860f0cc"
server_key_version: "encryptionkey@0"
I20260928 21:58:03.102003 20820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:58:03.112886 20820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:58:03.114902 20820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:58:03.115046 20820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:58:03.119004 20820 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.126:45007
I20260928 21:58:03.119083 20887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.126:45007 every 8 connection(s)
I20260928 21:58:03.119454 20820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
I20260928 21:58:03.120383 20888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:58:03.122860 20888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20e375dff94b4f598f1d4bdce221ca0b: Bootstrap starting.
I20260928 21:58:03.123426 20888 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20e375dff94b4f598f1d4bdce221ca0b: Neither blocks nor log segments found. Creating new log.
I20260928 21:58:03.123656 20888 log.cc:824] T 00000000000000000000000000000000 P 20e375dff94b4f598f1d4bdce221ca0b: Log is configured to *not* fsync() on all Append() calls
I20260928 21:58:03.124320 20888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20e375dff94b4f598f1d4bdce221ca0b: No bootstrap required, opened a new log
I20260928 21:58:03.125931 20888 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20e375dff94b4f598f1d4bdce221ca0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20e375dff94b4f598f1d4bdce221ca0b" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 45007 } }
I20260928 21:58:03.126083 20888 raft_consensus.cc:374] T 00000000000000000000000000000000 P 20e375dff94b4f598f1d4bdce221ca0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:58:03.126106 20888 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20e375dff94b4f598f1d4bdce221ca0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20e375dff94b4f598f1d4bdce221ca0b, State: Initialized, Role: FOLLOWER
I20260928 21:58:03.126215 20888 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20e375dff94b4f598f1d4bdce221ca0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20e375dff94b4f598f1d4bdce221ca0b" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 45007 } }
I20260928 21:58:03.126291 20888 raft_consensus.cc:388] T 00000000000000000000000000000000 P 20e375dff94b4f598f1d4bdce221ca0b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:58:03.126317 20888 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20e375dff94b4f598f1d4bdce221ca0b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:58:03.126343 20888 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20e375dff94b4f598f1d4bdce221ca0b [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:58:03.126866 20888 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20e375dff94b4f598f1d4bdce221ca0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20e375dff94b4f598f1d4bdce221ca0b" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 45007 } }
I20260928 21:58:03.126948 20888 leader_election.cc:302] T 00000000000000000000000000000000 P 20e375dff94b4f598f1d4bdce221ca0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 20e375dff94b4f598f1d4bdce221ca0b; no voters:
I20260928 21:58:03.127076 20888 leader_election.cc:288] T 00000000000000000000000000000000 P 20e375dff94b4f598f1d4bdce221ca0b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:58:03.127127 20891 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 20e375dff94b4f598f1d4bdce221ca0b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:58:03.127247 20891 raft_consensus.cc:686] T 00000000000000000000000000000000 P 20e375dff94b4f598f1d4bdce221ca0b [term 1 LEADER]: Becoming Leader. State: Replica: 20e375dff94b4f598f1d4bdce221ca0b, State: Running, Role: LEADER
I20260928 21:58:03.127375 20888 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20e375dff94b4f598f1d4bdce221ca0b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:58:03.127324 20891 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20e375dff94b4f598f1d4bdce221ca0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20e375dff94b4f598f1d4bdce221ca0b" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 45007 } }
I20260928 21:58:03.127823 20893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20e375dff94b4f598f1d4bdce221ca0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20e375dff94b4f598f1d4bdce221ca0b. Latest consensus state: current_term: 1 leader_uuid: "20e375dff94b4f598f1d4bdce221ca0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20e375dff94b4f598f1d4bdce221ca0b" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 45007 } } }
I20260928 21:58:03.127926 20893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20e375dff94b4f598f1d4bdce221ca0b [sys.catalog]: This master's current role is: LEADER
I20260928 21:58:03.127935 20892 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20e375dff94b4f598f1d4bdce221ca0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "20e375dff94b4f598f1d4bdce221ca0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20e375dff94b4f598f1d4bdce221ca0b" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 45007 } } }
I20260928 21:58:03.128054 20892 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20e375dff94b4f598f1d4bdce221ca0b [sys.catalog]: This master's current role is: LEADER
I20260928 21:58:03.128438 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 20820
I20260928 21:58:03.128562 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/minicluster-data/master-0/wal/instance
I20260928 21:58:03.128767 2777 external_mini_cluster.cc:1428] Setting key 8aeba80a49c3356c7a258a5309254a72
W20260928 21:58:03.129272 20907 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 20e375dff94b4f598f1d4bdce221ca0b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:58:03.129369 20907 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:58:03.129426 20900 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:58:03.129573 20900 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:58:03.131053 20900 catalog_manager.cc:1347] Generated new cluster ID: 0d839272d9c64dafa9b6633b4a3ebfdb
I20260928 21:58:03.131102 20900 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 21:58:03 dist-test-slave-k6lz krb5kdc[20807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:58:03.136965 20900 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:58:03.137611 20900 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:58:03.139925 20910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:03.129227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50272 (local address 127.2.182.126:45007)
0928 21:58:03.129481 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:03.129491 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:03.129526 (+ 35us) server_negotiation.cc:407] Connection header received
0928 21:58:03.129646 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:03.129653 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:03.129709 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:03.129826 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:03.130525 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:03.131946 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:03.132979 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:03.133626 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:03.133735 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:03.136484 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:03.136488 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:03.136489 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:03.136500 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:03.138379 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:03.138761 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:03.138764 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:03.138764 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:03.138876 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:03.139061 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:03.139062 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:03.139062 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:03.139300 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:03.139493 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:03.139559 (+ 66us) server_negotiation.cc:299] Negotiation successful
0928 21:58:03.139697 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":64,"threads_started":1}
I20260928 21:58:03.144609 20900 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 20e375dff94b4f598f1d4bdce221ca0b: Generated new TSK 0
I20260928 21:58:03.144793 20900 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.182.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab.
Entry for principal kudu/127.2.182.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab.
I20260928 21:58:03.172600 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.65:0
--local_ip_for_outbound_sockets=127.2.182.65
--webserver_interface=127.2.182.65
--webserver_port=0
--tserver_master_addrs=127.2.182.126:45007
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab
--principal=kudu/127.2.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:43241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:58:03.253309 20914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:58:03.253535 20914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:58:03.253568 20914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:58:03.254853 20914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:58:03.254904 20914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:58:03.254942 20914 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.65
I20260928 21:58:03.256505 20914 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:43241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab
--principal=kudu/127.2.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.182.65
--webserver_port=0
--tserver_master_addrs=127.2.182.126:45007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20914
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.65
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:58:03.256744 20914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:58:03.256959 20914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:58:03.261729 20919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:03.261723 20920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:03.261919 20914 server_base.cc:1034] running on GCE node
W20260928 21:58:03.261752 20922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:03.262158 20914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:58:03.262449 20914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:58:03.263592 20914 hybrid_clock.cc:648] HybridClock initialized: now 1790632683263578 us; error 24 us; skew 500 ppm
Sep 28 21:58:03 dist-test-slave-k6lz krb5kdc[20807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632683, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:58:03.266147 20914 init.cc:377] Logged in from keytab as kudu/127.2.182.65@KRBTEST.COM (short username kudu)
I20260928 21:58:03.266604 20914 webserver.cc:460] Webserver started at http://127.2.182.65:37047/ using document root <none> and password file <none>
I20260928 21:58:03.266803 20914 fs_manager.cc:359] Metadata directory not provided
I20260928 21:58:03.266849 20914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:58:03.267000 20914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:58:03.268056 20914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/minicluster-data/ts-0/data/instance:
uuid: "b1d004aff7074d71befee1b6a4a904ad"
format_stamp: "Formatted at 2026-09-28 21:58:03 on dist-test-slave-k6lz"
server_key: "dd6578dc73019f1e6291f6997319b977"
server_key_iv: "c910b5f06f83b9d22391c9b2e5be3aa6"
server_key_version: "encryptionkey@0"
I20260928 21:58:03.268374 20914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/minicluster-data/ts-0/wal/instance:
uuid: "b1d004aff7074d71befee1b6a4a904ad"
format_stamp: "Formatted at 2026-09-28 21:58:03 on dist-test-slave-k6lz"
server_key: "dd6578dc73019f1e6291f6997319b977"
server_key_iv: "c910b5f06f83b9d22391c9b2e5be3aa6"
server_key_version: "encryptionkey@0"
I20260928 21:58:03.269704 20914 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:58:03.270398 20929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:03.270565 20914 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:58:03.270645 20914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/minicluster-data/ts-0/wal
uuid: "b1d004aff7074d71befee1b6a4a904ad"
format_stamp: "Formatted at 2026-09-28 21:58:03 on dist-test-slave-k6lz"
server_key: "dd6578dc73019f1e6291f6997319b977"
server_key_iv: "c910b5f06f83b9d22391c9b2e5be3aa6"
server_key_version: "encryptionkey@0"
I20260928 21:58:03.270704 20914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:58:03.292929 20914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:58:03.295236 20914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:58:03.295421 20914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:58:03.295660 20914 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:58:03.295984 20914 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:58:03.296023 20914 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:03.296052 20914 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:58:03.296077 20914 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:03.301560 20914 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.65:40845
I20260928 21:58:03.301637 21042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.65:40845 every 8 connection(s)
I20260928 21:58:03.301939 20914 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
I20260928 21:58:03.308216 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 20914
I20260928 21:58:03.308317 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/minicluster-data/ts-0/wal/instance
I20260928 21:58:03.308568 2777 external_mini_cluster.cc:1428] Setting key f74f52f6592bb53448bbdcb35933935d
Sep 28 21:58:03 dist-test-slave-k6lz krb5kdc[20807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632683, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.65@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:58:03.312388 20910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:03.302678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.65:60649 (local address 127.2.182.126:45007)
0928 21:58:03.302747 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:03.302749 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:03.302932 (+ 183us) server_negotiation.cc:407] Connection header received
0928 21:58:03.303568 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:03.303569 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:03.303579 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:03.303636 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:03.304722 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:03.305368 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:03.306516 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:03.306719 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:03.306776 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:03.309165 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:03.309167 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:03.309167 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:03.309175 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:03.310858 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:03.311308 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:03.311310 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:03.311310 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:03.311403 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:03.311651 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:03.311652 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:03.311652 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:03.311775 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:03.311884 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:03.312199 (+ 315us) server_negotiation.cc:299] Negotiation successful
0928 21:58:03.312296 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 21:58:03.312564 21045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:03.302638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:45007 (local address 127.2.182.65:60649)
0928 21:58:03.302812 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:03.302850 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:03.303389 (+ 539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:03.303738 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:03.303742 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:03.304375 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:58:03.304587 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:03.304593 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:03.305458 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:03.305459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:03.306419 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:03.306422 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:03.306789 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:03.306790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:03.306896 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:03.307423 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:58:03.307431 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:58:03.309055 (+ 1624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:58:03.310994 (+ 1939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:03.311003 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:03.311005 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:03.311237 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:03.311475 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:03.311476 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:03.311476 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:03.311591 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:03.311891 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:58:03.311898 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:58:03.312040 (+ 142us) client_negotiation.cc:769] Sending connection context
0928 21:58:03.312154 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 21:58:03.312338 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":72,"threads_started":1}
I20260928 21:58:03.313274 21043 heartbeater.cc:347] Connected to a master server at 127.2.182.126:45007
I20260928 21:58:03.313392 21043 heartbeater.cc:464] Registering TS with master...
I20260928 21:58:03.313596 21043 heartbeater.cc:511] Master 127.2.182.126:45007 requested a full tablet report, sending...
I20260928 21:58:03.314198 20852 ts_manager.cc:194] Registered new tserver with Master: b1d004aff7074d71befee1b6a4a904ad (127.2.182.65:40845)
I20260928 21:58:03.315127 20852 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.65@KRBTEST.COM'} at 127.2.182.65:60649
WARNING: no policy specified for kudu/127.2.182.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab.
Entry for principal kudu/127.2.182.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab.
I20260928 21:58:03.336374 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.66:0
--local_ip_for_outbound_sockets=127.2.182.66
--webserver_interface=127.2.182.66
--webserver_port=0
--tserver_master_addrs=127.2.182.126:45007
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab
--principal=kudu/127.2.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:43241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:58:03.417150 21048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:58:03.417373 21048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:58:03.417405 21048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:58:03.418553 21048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:58:03.418594 21048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:58:03.418628 21048 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.66
I20260928 21:58:03.420194 21048 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:43241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab
--principal=kudu/127.2.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.182.66
--webserver_port=0
--tserver_master_addrs=127.2.182.126:45007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21048
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.66
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:58:03.420431 21048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:58:03.420642 21048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:58:03.425724 21054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:03.425715 21053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:03.425748 21048 server_base.cc:1034] running on GCE node
W20260928 21:58:03.425715 21056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:03.426076 21048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:58:03.426295 21048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:58:03.427387 21048 hybrid_clock.cc:648] HybridClock initialized: now 1790632683427368 us; error 34 us; skew 500 ppm
Sep 28 21:58:03 dist-test-slave-k6lz krb5kdc[20807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632683, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:58:03.430011 21048 init.cc:377] Logged in from keytab as kudu/127.2.182.66@KRBTEST.COM (short username kudu)
I20260928 21:58:03.430419 21048 webserver.cc:460] Webserver started at http://127.2.182.66:42651/ using document root <none> and password file <none>
I20260928 21:58:03.430624 21048 fs_manager.cc:359] Metadata directory not provided
I20260928 21:58:03.430672 21048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:58:03.430790 21048 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:58:03.431838 21048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/minicluster-data/ts-1/data/instance:
uuid: "6162b21b00cd482586cfb2748bed7ebe"
format_stamp: "Formatted at 2026-09-28 21:58:03 on dist-test-slave-k6lz"
server_key: "92b6fbc74828ff3f2c874e235b8c5bb4"
server_key_iv: "7fdc1828fdf3da756b4694237b5f7292"
server_key_version: "encryptionkey@0"
I20260928 21:58:03.432154 21048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/minicluster-data/ts-1/wal/instance:
uuid: "6162b21b00cd482586cfb2748bed7ebe"
format_stamp: "Formatted at 2026-09-28 21:58:03 on dist-test-slave-k6lz"
server_key: "92b6fbc74828ff3f2c874e235b8c5bb4"
server_key_iv: "7fdc1828fdf3da756b4694237b5f7292"
server_key_version: "encryptionkey@0"
I20260928 21:58:03.433434 21048 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:58:03.434150 21063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:03.434304 21048 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:58:03.434403 21048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/minicluster-data/ts-1/wal
uuid: "6162b21b00cd482586cfb2748bed7ebe"
format_stamp: "Formatted at 2026-09-28 21:58:03 on dist-test-slave-k6lz"
server_key: "92b6fbc74828ff3f2c874e235b8c5bb4"
server_key_iv: "7fdc1828fdf3da756b4694237b5f7292"
server_key_version: "encryptionkey@0"
I20260928 21:58:03.434463 21048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:58:03.454653 21048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:58:03.456730 21048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:58:03.456869 21048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:58:03.457090 21048 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:58:03.457377 21048 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:58:03.457408 21048 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:03.457438 21048 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:58:03.457461 21048 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:03.463039 21048 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.66:42345
I20260928 21:58:03.463095 21176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.66:42345 every 8 connection(s)
I20260928 21:58:03.463404 21048 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
Sep 28 21:58:03 dist-test-slave-k6lz krb5kdc[20807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632683, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.66@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:58:03.471218 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 21048
I20260928 21:58:03.471323 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/minicluster-data/ts-1/wal/instance
I20260928 21:58:03.471578 2777 external_mini_cluster.cc:1428] Setting key b89cd1ed6202d51506ad640971a6719e
I20260928 21:58:03.472558 20910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:03.464101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.66:38591 (local address 127.2.182.126:45007)
0928 21:58:03.464160 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:03.464162 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:03.464335 (+ 173us) server_negotiation.cc:407] Connection header received
0928 21:58:03.464809 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:03.464810 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:03.464820 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:03.464878 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:03.465833 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:03.466334 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:03.467506 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:03.467703 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:03.467752 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:03.469855 (+ 2103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:03.469858 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:03.469858 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:03.469866 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:03.471080 (+ 1214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:03.471561 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:03.471561 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:03.471562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:03.471618 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:03.471862 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:03.471863 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:03.471863 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:03.471999 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:03.472062 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:03.472401 (+ 339us) server_negotiation.cc:299] Negotiation successful
0928 21:58:03.472492 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 21:58:03.472837 21179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:03.464056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:45007 (local address 127.2.182.66:38591)
0928 21:58:03.464215 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:03.464257 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:03.464733 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:03.464934 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:03.464937 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:03.465518 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:58:03.465716 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:03.465721 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:03.466498 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:03.466499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:03.467314 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:03.467317 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:03.467790 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:03.467791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:03.467899 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:03.468382 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:58:03.468390 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:58:03.469751 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:58:03.471161 (+ 1410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:03.471169 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:03.471172 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:03.471476 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:03.471684 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:03.471685 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:03.471685 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:03.471797 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:03.472080 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:58:03.472086 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:58:03.472251 (+ 165us) client_negotiation.cc:769] Sending connection context
0928 21:58:03.472357 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 21:58:03.472613 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":87,"threads_started":1}
I20260928 21:58:03.473346 21177 heartbeater.cc:347] Connected to a master server at 127.2.182.126:45007
I20260928 21:58:03.473465 21177 heartbeater.cc:464] Registering TS with master...
I20260928 21:58:03.473656 21177 heartbeater.cc:511] Master 127.2.182.126:45007 requested a full tablet report, sending...
I20260928 21:58:03.474015 20852 ts_manager.cc:194] Registered new tserver with Master: 6162b21b00cd482586cfb2748bed7ebe (127.2.182.66:42345)
I20260928 21:58:03.474427 20852 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.66@KRBTEST.COM'} at 127.2.182.66:38591
WARNING: no policy specified for kudu/127.2.182.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab.
Entry for principal kudu/127.2.182.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab.
I20260928 21:58:03.498728 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.67:0
--local_ip_for_outbound_sockets=127.2.182.67
--webserver_interface=127.2.182.67
--webserver_port=0
--tserver_master_addrs=127.2.182.126:45007
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab
--principal=kudu/127.2.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:43241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:58:03.579186 21182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:58:03.579487 21182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:58:03.579528 21182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:58:03.581141 21182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:58:03.581195 21182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:58:03.581234 21182 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.67
I20260928 21:58:03.582888 21182 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:43241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab
--principal=kudu/127.2.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.182.67
--webserver_port=0
--tserver_master_addrs=127.2.182.126:45007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21182
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.67
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:58:03.583122 21182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:58:03.583377 21182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:58:03.588439 21188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:03.588609 21182 server_base.cc:1034] running on GCE node
W20260928 21:58:03.588430 21187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:03.588438 21190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:03.588900 21182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:58:03.589146 21182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:58:03.590293 21182 hybrid_clock.cc:648] HybridClock initialized: now 1790632683590275 us; error 33 us; skew 500 ppm
Sep 28 21:58:03 dist-test-slave-k6lz krb5kdc[20807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632683, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:58:03.592792 21182 init.cc:377] Logged in from keytab as kudu/127.2.182.67@KRBTEST.COM (short username kudu)
I20260928 21:58:03.593199 21182 webserver.cc:460] Webserver started at http://127.2.182.67:40759/ using document root <none> and password file <none>
I20260928 21:58:03.593391 21182 fs_manager.cc:359] Metadata directory not provided
I20260928 21:58:03.593437 21182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:58:03.593566 21182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:58:03.594595 21182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/minicluster-data/ts-2/data/instance:
uuid: "a5f70220b33541c29854d94f0fbb62f6"
format_stamp: "Formatted at 2026-09-28 21:58:03 on dist-test-slave-k6lz"
server_key: "598836acab490e6c61ae6b18c21944d8"
server_key_iv: "3309b62b57b64168bb598cf78a32d007"
server_key_version: "encryptionkey@0"
I20260928 21:58:03.594913 21182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/minicluster-data/ts-2/wal/instance:
uuid: "a5f70220b33541c29854d94f0fbb62f6"
format_stamp: "Formatted at 2026-09-28 21:58:03 on dist-test-slave-k6lz"
server_key: "598836acab490e6c61ae6b18c21944d8"
server_key_iv: "3309b62b57b64168bb598cf78a32d007"
server_key_version: "encryptionkey@0"
I20260928 21:58:03.596216 21182 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:58:03.597148 21197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:03.597362 21182 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:58:03.597453 21182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/minicluster-data/ts-2/wal
uuid: "a5f70220b33541c29854d94f0fbb62f6"
format_stamp: "Formatted at 2026-09-28 21:58:03 on dist-test-slave-k6lz"
server_key: "598836acab490e6c61ae6b18c21944d8"
server_key_iv: "3309b62b57b64168bb598cf78a32d007"
server_key_version: "encryptionkey@0"
I20260928 21:58:03.597528 21182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:58:03.619828 21182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:58:03.621980 21182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:58:03.622110 21182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:58:03.622323 21182 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:58:03.622609 21182 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:58:03.622642 21182 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:03.622671 21182 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:58:03.622695 21182 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:03.628189 21182 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.67:37913
I20260928 21:58:03.628312 21310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.67:37913 every 8 connection(s)
I20260928 21:58:03.628592 21182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
I20260928 21:58:03.634047 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 21182
I20260928 21:58:03.634158 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/minicluster-data/ts-2/wal/instance
I20260928 21:58:03.634333 2777 external_mini_cluster.cc:1428] Setting key 73a21c86816324464b844132e8336ef2
Sep 28 21:58:03 dist-test-slave-k6lz krb5kdc[20807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632683, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.67@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:58:03.638170 20910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:03.629278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.67:41005 (local address 127.2.182.126:45007)
0928 21:58:03.629318 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:03.629320 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:03.629532 (+ 212us) server_negotiation.cc:407] Connection header received
0928 21:58:03.629973 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:03.629974 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:03.629986 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:03.630061 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:03.630969 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:03.631591 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:03.632655 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:03.632852 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:03.632906 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:03.635136 (+ 2230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:03.635137 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:03.635137 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:03.635145 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:03.636473 (+ 1328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:03.637014 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:03.637015 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:03.637016 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:03.637177 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:03.637510 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:03.637511 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:03.637511 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:03.637640 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:03.637706 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:03.638034 (+ 328us) server_negotiation.cc:299] Negotiation successful
0928 21:58:03.638104 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 21:58:03.638485 21313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:03.629274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:45007 (local address 127.2.182.67:41005)
0928 21:58:03.629403 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:03.629438 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:03.629901 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:03.630112 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:03.630115 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:03.630664 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:58:03.630855 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:03.630860 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:03.631702 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:03.631704 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:03.632569 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:03.632573 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:03.632916 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:03.632917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:03.633014 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:03.633447 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:58:03.633454 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:58:03.634998 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:58:03.636608 (+ 1610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:03.636615 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:03.636617 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:03.636838 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:03.637252 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:03.637252 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:03.637253 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:03.637367 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:03.637747 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:58:03.637753 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:58:03.637883 (+ 130us) client_negotiation.cc:769] Sending connection context
0928 21:58:03.637984 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 21:58:03.638118 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":66,"threads_started":1}
I20260928 21:58:03.639011 21311 heartbeater.cc:347] Connected to a master server at 127.2.182.126:45007
I20260928 21:58:03.639129 21311 heartbeater.cc:464] Registering TS with master...
I20260928 21:58:03.639273 21311 heartbeater.cc:511] Master 127.2.182.126:45007 requested a full tablet report, sending...
I20260928 21:58:03.639644 20852 ts_manager.cc:194] Registered new tserver with Master: a5f70220b33541c29854d94f0fbb62f6 (127.2.182.67:37913)
I20260928 21:58:03.640093 20852 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.67@KRBTEST.COM'} at 127.2.182.67:41005
I20260928 21:58:03.646302 2777 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:58:03.653923 20910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:03.647639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50284 (local address 127.2.182.126:45007)
0928 21:58:03.647699 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:03.647701 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:03.647775 (+ 74us) server_negotiation.cc:407] Connection header received
0928 21:58:03.647820 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:03.647820 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:03.647829 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:03.647873 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:03.648370 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:03.648900 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:03.649726 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:03.649911 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:03.649963 (+ 52us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:03.651189 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:03.651191 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:03.651191 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:03.651198 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:03.652447 (+ 1249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:03.652840 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:03.652841 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:03.652841 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:03.652894 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:03.653138 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:03.653139 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:03.653139 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:03.653364 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:03.653459 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:03.653751 (+ 292us) server_negotiation.cc:299] Negotiation successful
0928 21:58:03.653817 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 21:58:03.655671 20852 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50284:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:58:03.657232 20852 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:58:03.663552 21325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:03.659682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.66:42345 (local address 127.0.0.1:60362)
0928 21:58:03.660052 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:03.660061 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:03.660098 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:03.660333 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:03.660334 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:03.660337 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:58:03.660386 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:03.660388 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:03.661310 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:03.661311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:03.662304 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:03.662306 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:03.663199 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:03.663200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:03.663282 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:03.663296 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 21:58:03.663411 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 21:58:03.663464 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"mutex_wait_us":3,"thread_start_us":92,"threads_started":1}
I20260928 21:58:03.663553 21326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:03.660000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60362 (local address 127.2.182.66:42345)
0928 21:58:03.660171 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:03.660176 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:03.660188 (+ 12us) server_negotiation.cc:407] Connection header received
0928 21:58:03.660227 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:03.660230 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:03.660253 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:03.660343 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:58:03.660433 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:03.661217 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:03.662395 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:03.663113 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:03.663185 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:03.663243 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:03.663426 (+ 183us) server_negotiation.cc:299] Negotiation successful
0928 21:58:03.663468 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":70,"threads_started":1}
I20260928 21:58:03.663820 21323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:03.659674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.65:40845 (local address 127.0.0.1:37652)
0928 21:58:03.659829 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:03.659868 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:03.659938 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:03.660159 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:03.660161 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:03.660163 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:58:03.660238 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:03.660240 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:03.661324 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:03.661325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:03.662657 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:03.662661 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:03.663634 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:03.663636 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:03.663702 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:03.663706 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:58:03.663734 (+ 28us) client_negotiation.cc:240] Negotiation successful
0928 21:58:03.663770 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":9,"thread_start_us":64,"threads_started":1}
I20260928 21:58:03.664165 21324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:03.659792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37652 (local address 127.2.182.65:40845)
0928 21:58:03.659984 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:03.659989 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:03.659999 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:58:03.660039 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:03.660042 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:03.660057 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:03.660317 (+ 260us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:58:03.660389 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:03.661229 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:03.662743 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:03.663529 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:03.663938 (+ 409us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:03.664023 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:03.664048 (+ 25us) server_negotiation.cc:299] Negotiation successful
0928 21:58:03.664091 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":86,"threads_started":1}
I20260928 21:58:03.664469 21111 tablet_service.cc:1467] Processing CreateTablet for tablet 7ca2e2c04ee442f1a1facd4f6997bd22 (DEFAULT_TABLE table=test-table [id=0aded74669d04dfd9e192a3eb8a71067]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:58:03.664796 21111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ca2e2c04ee442f1a1facd4f6997bd22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:58:03.664952 21322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:03.659938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.67:37913 (local address 127.0.0.1:57980)
0928 21:58:03.661346 (+ 1408us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:03.661357 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:03.661410 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:03.661588 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:03.661589 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:03.661591 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:58:03.661627 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:03.661628 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:03.662787 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:03.662787 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:03.663666 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:03.663669 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:03.664685 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:03.664686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:03.664774 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:03.664778 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:58:03.664845 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 21:58:03.664887 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1384,"thread_start_us":82,"threads_started":1}
I20260928 21:58:03.665163 21327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:03.660149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57980 (local address 127.2.182.67:37913)
0928 21:58:03.660322 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:03.660326 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:03.661426 (+ 1100us) server_negotiation.cc:407] Connection header received
0928 21:58:03.661476 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:03.661479 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:03.661495 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:03.661579 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:58:03.661680 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:03.662715 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:03.663772 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:03.664599 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:03.664673 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:03.664729 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:03.665048 (+ 319us) server_negotiation.cc:299] Negotiation successful
0928 21:58:03.665091 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":80,"threads_started":1}
I20260928 21:58:03.665944 21245 tablet_service.cc:1467] Processing CreateTablet for tablet 7ca2e2c04ee442f1a1facd4f6997bd22 (DEFAULT_TABLE table=test-table [id=0aded74669d04dfd9e192a3eb8a71067]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:58:03.666234 21245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ca2e2c04ee442f1a1facd4f6997bd22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:58:03.668320 21328 tablet_bootstrap.cc:492] T 7ca2e2c04ee442f1a1facd4f6997bd22 P 6162b21b00cd482586cfb2748bed7ebe: Bootstrap starting.
I20260928 21:58:03.668648 20977 tablet_service.cc:1467] Processing CreateTablet for tablet 7ca2e2c04ee442f1a1facd4f6997bd22 (DEFAULT_TABLE table=test-table [id=0aded74669d04dfd9e192a3eb8a71067]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:58:03.668855 21328 tablet_bootstrap.cc:654] T 7ca2e2c04ee442f1a1facd4f6997bd22 P 6162b21b00cd482586cfb2748bed7ebe: Neither blocks nor log segments found. Creating new log.
I20260928 21:58:03.668992 20977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ca2e2c04ee442f1a1facd4f6997bd22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:58:03.669358 21328 log.cc:824] T 7ca2e2c04ee442f1a1facd4f6997bd22 P 6162b21b00cd482586cfb2748bed7ebe: Log is configured to *not* fsync() on all Append() calls
I20260928 21:58:03.669659 21329 tablet_bootstrap.cc:492] T 7ca2e2c04ee442f1a1facd4f6997bd22 P a5f70220b33541c29854d94f0fbb62f6: Bootstrap starting.
I20260928 21:58:03.670028 21328 tablet_bootstrap.cc:492] T 7ca2e2c04ee442f1a1facd4f6997bd22 P 6162b21b00cd482586cfb2748bed7ebe: No bootstrap required, opened a new log
I20260928 21:58:03.670092 21328 ts_tablet_manager.cc:1397] T 7ca2e2c04ee442f1a1facd4f6997bd22 P 6162b21b00cd482586cfb2748bed7ebe: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:58:03.671532 21328 raft_consensus.cc:348] T 7ca2e2c04ee442f1a1facd4f6997bd22 P 6162b21b00cd482586cfb2748bed7ebe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6162b21b00cd482586cfb2748bed7ebe" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 42345 } } peers { permanent_uuid: "a5f70220b33541c29854d94f0fbb62f6" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 37913 } } peers { permanent_uuid: "b1d004aff7074d71befee1b6a4a904ad" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 40845 } }
I20260928 21:58:03.671711 21328 raft_consensus.cc:374] T 7ca2e2c04ee442f1a1facd4f6997bd22 P 6162b21b00cd482586cfb2748bed7ebe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:58:03.671756 21328 raft_consensus.cc:729] T 7ca2e2c04ee442f1a1facd4f6997bd22 P 6162b21b00cd482586cfb2748bed7ebe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6162b21b00cd482586cfb2748bed7ebe, State: Initialized, Role: FOLLOWER
I20260928 21:58:03.671875 21328 consensus_queue.cc:260] T 7ca2e2c04ee442f1a1facd4f6997bd22 P 6162b21b00cd482586cfb2748bed7ebe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6162b21b00cd482586cfb2748bed7ebe" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 42345 } } peers { permanent_uuid: "a5f70220b33541c29854d94f0fbb62f6" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 37913 } } peers { permanent_uuid: "b1d004aff7074d71befee1b6a4a904ad" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 40845 } }
I20260928 21:58:03.671967 21329 tablet_bootstrap.cc:654] T 7ca2e2c04ee442f1a1facd4f6997bd22 P a5f70220b33541c29854d94f0fbb62f6: Neither blocks nor log segments found. Creating new log.
I20260928 21:58:03.672163 21328 ts_tablet_manager.cc:1428] T 7ca2e2c04ee442f1a1facd4f6997bd22 P 6162b21b00cd482586cfb2748bed7ebe: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 21:58:03.672174 21177 heartbeater.cc:503] Master 127.2.182.126:45007 was elected leader, sending a full tablet report...
I20260928 21:58:03.672494 21329 log.cc:824] T 7ca2e2c04ee442f1a1facd4f6997bd22 P a5f70220b33541c29854d94f0fbb62f6: Log is configured to *not* fsync() on all Append() calls
I20260928 21:58:03.672958 21331 tablet_bootstrap.cc:492] T 7ca2e2c04ee442f1a1facd4f6997bd22 P b1d004aff7074d71befee1b6a4a904ad: Bootstrap starting.
I20260928 21:58:03.673391 21329 tablet_bootstrap.cc:492] T 7ca2e2c04ee442f1a1facd4f6997bd22 P a5f70220b33541c29854d94f0fbb62f6: No bootstrap required, opened a new log
I20260928 21:58:03.673471 21329 ts_tablet_manager.cc:1397] T 7ca2e2c04ee442f1a1facd4f6997bd22 P a5f70220b33541c29854d94f0fbb62f6: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 21:58:03.674901 21329 raft_consensus.cc:348] T 7ca2e2c04ee442f1a1facd4f6997bd22 P a5f70220b33541c29854d94f0fbb62f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6162b21b00cd482586cfb2748bed7ebe" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 42345 } } peers { permanent_uuid: "a5f70220b33541c29854d94f0fbb62f6" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 37913 } } peers { permanent_uuid: "b1d004aff7074d71befee1b6a4a904ad" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 40845 } }
I20260928 21:58:03.675048 21329 raft_consensus.cc:374] T 7ca2e2c04ee442f1a1facd4f6997bd22 P a5f70220b33541c29854d94f0fbb62f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:58:03.675079 21329 raft_consensus.cc:729] T 7ca2e2c04ee442f1a1facd4f6997bd22 P a5f70220b33541c29854d94f0fbb62f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5f70220b33541c29854d94f0fbb62f6, State: Initialized, Role: FOLLOWER
I20260928 21:58:03.675195 21329 consensus_queue.cc:260] T 7ca2e2c04ee442f1a1facd4f6997bd22 P a5f70220b33541c29854d94f0fbb62f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6162b21b00cd482586cfb2748bed7ebe" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 42345 } } peers { permanent_uuid: "a5f70220b33541c29854d94f0fbb62f6" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 37913 } } peers { permanent_uuid: "b1d004aff7074d71befee1b6a4a904ad" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 40845 } }
I20260928 21:58:03.675515 21311 heartbeater.cc:503] Master 127.2.182.126:45007 was elected leader, sending a full tablet report...
I20260928 21:58:03.675498 21329 ts_tablet_manager.cc:1428] T 7ca2e2c04ee442f1a1facd4f6997bd22 P a5f70220b33541c29854d94f0fbb62f6: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 21:58:03.675951 21331 tablet_bootstrap.cc:654] T 7ca2e2c04ee442f1a1facd4f6997bd22 P b1d004aff7074d71befee1b6a4a904ad: Neither blocks nor log segments found. Creating new log.
I20260928 21:58:03.676462 21331 log.cc:824] T 7ca2e2c04ee442f1a1facd4f6997bd22 P b1d004aff7074d71befee1b6a4a904ad: Log is configured to *not* fsync() on all Append() calls
I20260928 21:58:03.677181 21331 tablet_bootstrap.cc:492] T 7ca2e2c04ee442f1a1facd4f6997bd22 P b1d004aff7074d71befee1b6a4a904ad: No bootstrap required, opened a new log
I20260928 21:58:03.677251 21331 ts_tablet_manager.cc:1397] T 7ca2e2c04ee442f1a1facd4f6997bd22 P b1d004aff7074d71befee1b6a4a904ad: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260928 21:58:03.678655 21331 raft_consensus.cc:348] T 7ca2e2c04ee442f1a1facd4f6997bd22 P b1d004aff7074d71befee1b6a4a904ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6162b21b00cd482586cfb2748bed7ebe" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 42345 } } peers { permanent_uuid: "a5f70220b33541c29854d94f0fbb62f6" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 37913 } } peers { permanent_uuid: "b1d004aff7074d71befee1b6a4a904ad" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 40845 } }
I20260928 21:58:03.678814 21331 raft_consensus.cc:374] T 7ca2e2c04ee442f1a1facd4f6997bd22 P b1d004aff7074d71befee1b6a4a904ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:58:03.678848 21331 raft_consensus.cc:729] T 7ca2e2c04ee442f1a1facd4f6997bd22 P b1d004aff7074d71befee1b6a4a904ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1d004aff7074d71befee1b6a4a904ad, State: Initialized, Role: FOLLOWER
I20260928 21:58:03.678973 21331 consensus_queue.cc:260] T 7ca2e2c04ee442f1a1facd4f6997bd22 P b1d004aff7074d71befee1b6a4a904ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6162b21b00cd482586cfb2748bed7ebe" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 42345 } } peers { permanent_uuid: "a5f70220b33541c29854d94f0fbb62f6" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 37913 } } peers { permanent_uuid: "b1d004aff7074d71befee1b6a4a904ad" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 40845 } }
I20260928 21:58:03.679246 21331 ts_tablet_manager.cc:1428] T 7ca2e2c04ee442f1a1facd4f6997bd22 P b1d004aff7074d71befee1b6a4a904ad: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 21:58:03.679270 21043 heartbeater.cc:503] Master 127.2.182.126:45007 was elected leader, sending a full tablet report...
I20260928 21:58:03.712869 21334 raft_consensus.cc:482] T 7ca2e2c04ee442f1a1facd4f6997bd22 P a5f70220b33541c29854d94f0fbb62f6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:58:03.713007 21334 raft_consensus.cc:504] T 7ca2e2c04ee442f1a1facd4f6997bd22 P a5f70220b33541c29854d94f0fbb62f6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6162b21b00cd482586cfb2748bed7ebe" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 42345 } } peers { permanent_uuid: "a5f70220b33541c29854d94f0fbb62f6" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 37913 } } peers { permanent_uuid: "b1d004aff7074d71befee1b6a4a904ad" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 40845 } }
I20260928 21:58:03.713378 21334 leader_election.cc:288] T 7ca2e2c04ee442f1a1facd4f6997bd22 P a5f70220b33541c29854d94f0fbb62f6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6162b21b00cd482586cfb2748bed7ebe (127.2.182.66:42345), b1d004aff7074d71befee1b6a4a904ad (127.2.182.65:40845)
W20260928 21:58:03.713912 21178 tablet.cc:2369] T 7ca2e2c04ee442f1a1facd4f6997bd22 P 6162b21b00cd482586cfb2748bed7ebe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:58:03.716253 21326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:03.713613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.67:43163 (local address 127.2.182.66:42345)
0928 21:58:03.713655 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:03.713658 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:03.713699 (+ 41us) server_negotiation.cc:407] Connection header received
0928 21:58:03.713788 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:03.713788 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:03.713799 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:03.713852 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:58:03.714028 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:03.714589 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:03.715572 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:03.715932 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:03.715994 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:03.716004 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:03.716147 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 21:58:03.716188 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 21:58:03.716253 21313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:03.713552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.66:42345 (local address 127.2.182.67:43163)
0928 21:58:03.713642 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:03.713655 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:03.713717 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:03.713867 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:03.713868 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:03.713870 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:58:03.713942 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:03.713944 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:03.714671 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:03.714672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:03.715481 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:03.715484 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:03.716007 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:03.716008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:03.716086 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:03.716089 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:58:03.716134 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 21:58:03.716175 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260928 21:58:03.716550 21324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:03.713693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.67:53883 (local address 127.2.182.65:40845)
0928 21:58:03.713757 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:03.713759 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:03.713978 (+ 219us) server_negotiation.cc:407] Connection header received
0928 21:58:03.714052 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:03.714053 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:03.714063 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:03.714090 (+ 27us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:58:03.714302 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:03.714831 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:03.715817 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:03.716203 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:03.716266 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:03.716274 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:03.716440 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 21:58:03.716485 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 21:58:03.716581 21131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ca2e2c04ee442f1a1facd4f6997bd22" candidate_uuid: "a5f70220b33541c29854d94f0fbb62f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6162b21b00cd482586cfb2748bed7ebe" is_pre_election: true
I20260928 21:58:03.716755 21337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:03.713578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.65:40845 (local address 127.2.182.67:53883)
0928 21:58:03.713905 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:03.713914 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:03.713970 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:03.714151 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:03.714151 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:03.714152 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:58:03.714209 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:03.714210 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:03.714940 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:03.714941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:03.715738 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:03.715740 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:03.716279 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:03.716280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:03.716352 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:03.716356 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:58:03.716642 (+ 286us) client_negotiation.cc:240] Negotiation successful
0928 21:58:03.716683 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":143,"thread_start_us":111,"threads_started":1}
I20260928 21:58:03.716753 21131 raft_consensus.cc:2454] T 7ca2e2c04ee442f1a1facd4f6997bd22 P 6162b21b00cd482586cfb2748bed7ebe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5f70220b33541c29854d94f0fbb62f6 in term 0.
I20260928 21:58:03.716972 21200 leader_election.cc:302] T 7ca2e2c04ee442f1a1facd4f6997bd22 P a5f70220b33541c29854d94f0fbb62f6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6162b21b00cd482586cfb2748bed7ebe, a5f70220b33541c29854d94f0fbb62f6; no voters:
I20260928 21:58:03.717106 21334 raft_consensus.cc:2781] T 7ca2e2c04ee442f1a1facd4f6997bd22 P a5f70220b33541c29854d94f0fbb62f6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:58:03.717077 20997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ca2e2c04ee442f1a1facd4f6997bd22" candidate_uuid: "a5f70220b33541c29854d94f0fbb62f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1d004aff7074d71befee1b6a4a904ad" is_pre_election: true
I20260928 21:58:03.717159 21334 raft_consensus.cc:482] T 7ca2e2c04ee442f1a1facd4f6997bd22 P a5f70220b33541c29854d94f0fbb62f6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:58:03.717177 21334 raft_consensus.cc:3037] T 7ca2e2c04ee442f1a1facd4f6997bd22 P a5f70220b33541c29854d94f0fbb62f6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:58:03.717213 20997 raft_consensus.cc:2454] T 7ca2e2c04ee442f1a1facd4f6997bd22 P b1d004aff7074d71befee1b6a4a904ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5f70220b33541c29854d94f0fbb62f6 in term 0.
I20260928 21:58:03.717958 21334 raft_consensus.cc:504] T 7ca2e2c04ee442f1a1facd4f6997bd22 P a5f70220b33541c29854d94f0fbb62f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6162b21b00cd482586cfb2748bed7ebe" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 42345 } } peers { permanent_uuid: "a5f70220b33541c29854d94f0fbb62f6" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 37913 } } peers { permanent_uuid: "b1d004aff7074d71befee1b6a4a904ad" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 40845 } }
I20260928 21:58:03.718072 21334 leader_election.cc:288] T 7ca2e2c04ee442f1a1facd4f6997bd22 P a5f70220b33541c29854d94f0fbb62f6 [CANDIDATE]: Term 1 election: Requested vote from peers 6162b21b00cd482586cfb2748bed7ebe (127.2.182.66:42345), b1d004aff7074d71befee1b6a4a904ad (127.2.182.65:40845)
I20260928 21:58:03.718245 20997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ca2e2c04ee442f1a1facd4f6997bd22" candidate_uuid: "a5f70220b33541c29854d94f0fbb62f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1d004aff7074d71befee1b6a4a904ad"
I20260928 21:58:03.718266 21131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ca2e2c04ee442f1a1facd4f6997bd22" candidate_uuid: "a5f70220b33541c29854d94f0fbb62f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6162b21b00cd482586cfb2748bed7ebe"
I20260928 21:58:03.718327 20997 raft_consensus.cc:3037] T 7ca2e2c04ee442f1a1facd4f6997bd22 P b1d004aff7074d71befee1b6a4a904ad [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:58:03.718327 21131 raft_consensus.cc:3037] T 7ca2e2c04ee442f1a1facd4f6997bd22 P 6162b21b00cd482586cfb2748bed7ebe [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:58:03.719168 21131 raft_consensus.cc:2454] T 7ca2e2c04ee442f1a1facd4f6997bd22 P 6162b21b00cd482586cfb2748bed7ebe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5f70220b33541c29854d94f0fbb62f6 in term 1.
I20260928 21:58:03.719169 20997 raft_consensus.cc:2454] T 7ca2e2c04ee442f1a1facd4f6997bd22 P b1d004aff7074d71befee1b6a4a904ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5f70220b33541c29854d94f0fbb62f6 in term 1.
I20260928 21:58:03.719377 21200 leader_election.cc:302] T 7ca2e2c04ee442f1a1facd4f6997bd22 P a5f70220b33541c29854d94f0fbb62f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6162b21b00cd482586cfb2748bed7ebe, a5f70220b33541c29854d94f0fbb62f6; no voters:
I20260928 21:58:03.719475 21334 raft_consensus.cc:2781] T 7ca2e2c04ee442f1a1facd4f6997bd22 P a5f70220b33541c29854d94f0fbb62f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:58:03.719630 21334 raft_consensus.cc:686] T 7ca2e2c04ee442f1a1facd4f6997bd22 P a5f70220b33541c29854d94f0fbb62f6 [term 1 LEADER]: Becoming Leader. State: Replica: a5f70220b33541c29854d94f0fbb62f6, State: Running, Role: LEADER
I20260928 21:58:03.719719 21334 consensus_queue.cc:237] T 7ca2e2c04ee442f1a1facd4f6997bd22 P a5f70220b33541c29854d94f0fbb62f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6162b21b00cd482586cfb2748bed7ebe" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 42345 } } peers { permanent_uuid: "a5f70220b33541c29854d94f0fbb62f6" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 37913 } } peers { permanent_uuid: "b1d004aff7074d71befee1b6a4a904ad" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 40845 } }
I20260928 21:58:03.720692 20852 catalog_manager.cc:5495] T 7ca2e2c04ee442f1a1facd4f6997bd22 P a5f70220b33541c29854d94f0fbb62f6 reported cstate change: term changed from 0 to 1, leader changed from <none> to a5f70220b33541c29854d94f0fbb62f6 (127.2.182.67). New cstate: current_term: 1 leader_uuid: "a5f70220b33541c29854d94f0fbb62f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6162b21b00cd482586cfb2748bed7ebe" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 42345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5f70220b33541c29854d94f0fbb62f6" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 37913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b1d004aff7074d71befee1b6a4a904ad" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 40845 } health_report { overall_health: UNKNOWN } } }
I20260928 21:58:03.740170 21327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:03.737603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57988 (local address 127.2.182.67:37913)
0928 21:58:03.737645 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:03.737647 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:03.737657 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:58:03.737747 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:03.737748 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:03.737770 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:03.737839 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:58:03.737944 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:03.738413 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:03.739241 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:03.739534 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:03.739602 (+ 68us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:03.739782 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:58:03.739876 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:58:03.739962 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:03.740058 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 21:58:03.740069 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 21:58:03.742110 20997 raft_consensus.cc:1261] T 7ca2e2c04ee442f1a1facd4f6997bd22 P b1d004aff7074d71befee1b6a4a904ad [term 1 FOLLOWER]: Refusing update from remote peer a5f70220b33541c29854d94f0fbb62f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:58:03.742136 21131 raft_consensus.cc:1261] T 7ca2e2c04ee442f1a1facd4f6997bd22 P 6162b21b00cd482586cfb2748bed7ebe [term 1 FOLLOWER]: Refusing update from remote peer a5f70220b33541c29854d94f0fbb62f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:58:03.742470 21338 consensus_queue.cc:1035] T 7ca2e2c04ee442f1a1facd4f6997bd22 P a5f70220b33541c29854d94f0fbb62f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1d004aff7074d71befee1b6a4a904ad" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 40845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:58:03.742612 21334 consensus_queue.cc:1035] T 7ca2e2c04ee442f1a1facd4f6997bd22 P a5f70220b33541c29854d94f0fbb62f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6162b21b00cd482586cfb2748bed7ebe" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 42345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:58:03.744968 21343 mvcc.cc:205] Tried to move back new op lower bound from 7334431472605134848 to 7334431472517087232. Current Snapshot: MvccSnapshot[applied={T|T < 7334431472605134848}]
I20260928 21:58:03.745088 21342 mvcc.cc:205] Tried to move back new op lower bound from 7334431472605134848 to 7334431472517087232. Current Snapshot: MvccSnapshot[applied={T|T < 7334431472605134848}]
I20260928 21:58:03.747243 20852 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50284:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:58:03.747360 20852 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50284:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:58:03.748061 20852 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 20e375dff94b4f598f1d4bdce221ca0b: Sending DeleteTablet for 3 replicas of tablet 7ca2e2c04ee442f1a1facd4f6997bd22
I20260928 21:58:03.748377 21111 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ca2e2c04ee442f1a1facd4f6997bd22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:58:03 UTC) from {username='kudu', principal='kudu/127.2.182.126@KRBTEST.COM'} at 127.0.0.1:60362
I20260928 21:58:03.748481 20977 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ca2e2c04ee442f1a1facd4f6997bd22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:58:03 UTC) from {username='kudu', principal='kudu/127.2.182.126@KRBTEST.COM'} at 127.0.0.1:37652
I20260928 21:58:03.748588 21351 tablet_replica.cc:330] T 7ca2e2c04ee442f1a1facd4f6997bd22 P 6162b21b00cd482586cfb2748bed7ebe: stopping tablet replica
I20260928 21:58:03.748649 21245 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ca2e2c04ee442f1a1facd4f6997bd22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:58:03 UTC) from {username='kudu', principal='kudu/127.2.182.126@KRBTEST.COM'} at 127.0.0.1:57980
I20260928 21:58:03.748680 21351 raft_consensus.cc:2229] T 7ca2e2c04ee442f1a1facd4f6997bd22 P 6162b21b00cd482586cfb2748bed7ebe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:58:03.748775 21351 raft_consensus.cc:2258] T 7ca2e2c04ee442f1a1facd4f6997bd22 P 6162b21b00cd482586cfb2748bed7ebe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:58:03.748849 21352 tablet_replica.cc:330] T 7ca2e2c04ee442f1a1facd4f6997bd22 P b1d004aff7074d71befee1b6a4a904ad: stopping tablet replica
I20260928 21:58:03.748871 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 20914
I20260928 21:58:03.748931 21352 raft_consensus.cc:2229] T 7ca2e2c04ee442f1a1facd4f6997bd22 P b1d004aff7074d71befee1b6a4a904ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:58:03.749015 21352 raft_consensus.cc:2258] T 7ca2e2c04ee442f1a1facd4f6997bd22 P b1d004aff7074d71befee1b6a4a904ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:58:03.749399 21353 tablet_replica.cc:330] T 7ca2e2c04ee442f1a1facd4f6997bd22 P a5f70220b33541c29854d94f0fbb62f6: stopping tablet replica
I20260928 21:58:03.750095 21353 raft_consensus.cc:2229] T 7ca2e2c04ee442f1a1facd4f6997bd22 P a5f70220b33541c29854d94f0fbb62f6 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:58:03.750217 21353 raft_consensus.cc:2258] T 7ca2e2c04ee442f1a1facd4f6997bd22 P a5f70220b33541c29854d94f0fbb62f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:58:03.750605 21353 ts_tablet_manager.cc:1905] T 7ca2e2c04ee442f1a1facd4f6997bd22 P a5f70220b33541c29854d94f0fbb62f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:58:03.751137 21351 ts_tablet_manager.cc:1905] T 7ca2e2c04ee442f1a1facd4f6997bd22 P 6162b21b00cd482586cfb2748bed7ebe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:58:03.752398 21353 ts_tablet_manager.cc:1918] T 7ca2e2c04ee442f1a1facd4f6997bd22 P a5f70220b33541c29854d94f0fbb62f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:58:03.752485 21353 log.cc:1196] T 7ca2e2c04ee442f1a1facd4f6997bd22 P a5f70220b33541c29854d94f0fbb62f6: Deleting WAL directory at /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/minicluster-data/ts-2/wal/wals/7ca2e2c04ee442f1a1facd4f6997bd22
I20260928 21:58:03.752609 21351 ts_tablet_manager.cc:1918] T 7ca2e2c04ee442f1a1facd4f6997bd22 P 6162b21b00cd482586cfb2748bed7ebe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:58:03.752681 21351 log.cc:1196] T 7ca2e2c04ee442f1a1facd4f6997bd22 P 6162b21b00cd482586cfb2748bed7ebe: Deleting WAL directory at /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632440055392-2777-0/minicluster-data/ts-1/wal/wals/7ca2e2c04ee442f1a1facd4f6997bd22
I20260928 21:58:03.752740 21353 ts_tablet_manager.cc:1939] T 7ca2e2c04ee442f1a1facd4f6997bd22 P a5f70220b33541c29854d94f0fbb62f6: Deleting consensus metadata
I20260928 21:58:03.752928 21351 ts_tablet_manager.cc:1939] T 7ca2e2c04ee442f1a1facd4f6997bd22 P 6162b21b00cd482586cfb2748bed7ebe: Deleting consensus metadata
I20260928 21:58:03.753269 20838 catalog_manager.cc:4841] TS a5f70220b33541c29854d94f0fbb62f6 (127.2.182.67:37913): tablet 7ca2e2c04ee442f1a1facd4f6997bd22 (table test-table [id=0aded74669d04dfd9e192a3eb8a71067]) successfully deleted
I20260928 21:58:03.753379 20838 catalog_manager.cc:4841] TS 6162b21b00cd482586cfb2748bed7ebe (127.2.182.66:42345): tablet 7ca2e2c04ee442f1a1facd4f6997bd22 (table test-table [id=0aded74669d04dfd9e192a3eb8a71067]) successfully deleted
W20260928 21:58:03.756623 20839 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.182.65:40845 (error 108)
I20260928 21:58:03.757196 21322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:03.757074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.65:40845 (local address 127.0.0.1:37656)
0928 21:58:03.757127 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:03.757165 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.65:40845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 21:58:03.757308 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 21048
W20260928 21:58:03.757344 20839 catalog_manager.cc:4584] TS b1d004aff7074d71befee1b6a4a904ad (127.2.182.65:40845): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7ca2e2c04ee442f1a1facd4f6997bd22: Network error: Client connection negotiation failed: client connection to 127.2.182.65:40845: connect: Connection refused (error 111)
I20260928 21:58:03.763656 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 21182
I20260928 21:58:03.768396 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 20820
2026-09-28T21:58:03Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1845 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-28T21:58:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:58:03Z Disabled control of system clock
I20260928 21:58:03.794651 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632440055392-2777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632440055392-2777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632440055392-2777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:39553
--webserver_interface=127.2.182.126
--webserver_port=0
--builtin_ntp_servers=127.2.182.84:37207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.126:39553
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 21:58:03.876010 21358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:58:03.876228 21358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:58:03.876250 21358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:58:03.877378 21358 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:58:03.877415 21358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:58:03.877427 21358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:58:03.877439 21358 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:58:03.877449 21358 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:58:03.878808 21358 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:37207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632440055392-2777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632440055392-2777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.126:39553
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:39553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21358
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632440055392-2777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:58:03.879019 21358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:58:03.879257 21358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:58:03.884157 21363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:03.884222 21364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:03.884179 21358 server_base.cc:1034] running on GCE node
W20260928 21:58:03.884390 21366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:03.884629 21358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:58:03.884912 21358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:58:03.886096 21358 hybrid_clock.cc:648] HybridClock initialized: now 1790632683886084 us; error 28 us; skew 500 ppm
I20260928 21:58:03.887398 21358 webserver.cc:460] Webserver started at http://127.2.182.126:41835/ using document root <none> and password file <none>
I20260928 21:58:03.887621 21358 fs_manager.cc:359] Metadata directory not provided
I20260928 21:58:03.887669 21358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:58:03.887791 21358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:58:03.888870 21358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632440055392-2777-0/minicluster-data/master-0/data/instance:
uuid: "66db1dbf2cdb405b85b7281ea8d73220"
format_stamp: "Formatted at 2026-09-28 21:58:03 on dist-test-slave-k6lz"
server_key: "dc2bfcf7a805e64e05ebefe3d8c11270"
server_key_iv: "d8ab340d457233543a6222e19de4e6d6"
server_key_version: "encryptionkey@0"
I20260928 21:58:03.889214 21358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632440055392-2777-0/minicluster-data/master-0/wal/instance:
uuid: "66db1dbf2cdb405b85b7281ea8d73220"
format_stamp: "Formatted at 2026-09-28 21:58:03 on dist-test-slave-k6lz"
server_key: "dc2bfcf7a805e64e05ebefe3d8c11270"
server_key_iv: "d8ab340d457233543a6222e19de4e6d6"
server_key_version: "encryptionkey@0"
I20260928 21:58:03.890659 21358 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 21:58:03.891641 21372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:03.891855 21358 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:58:03.891978 21358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632440055392-2777-0/minicluster-data/master-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632440055392-2777-0/minicluster-data/master-0/wal
uuid: "66db1dbf2cdb405b85b7281ea8d73220"
format_stamp: "Formatted at 2026-09-28 21:58:03 on dist-test-slave-k6lz"
server_key: "dc2bfcf7a805e64e05ebefe3d8c11270"
server_key_iv: "d8ab340d457233543a6222e19de4e6d6"
server_key_version: "encryptionkey@0"
I20260928 21:58:03.892046 21358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632440055392-2777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632440055392-2777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632440055392-2777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:58:03.903157 21358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:58:03.903512 21358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:58:03.903630 21358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:58:03.907439 21358 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.126:39553
I20260928 21:58:03.907531 21424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.126:39553 every 8 connection(s)
I20260928 21:58:03.907837 21358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
I20260928 21:58:03.908767 21425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:58:03.909124 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 21358
I20260928 21:58:03.909217 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632440055392-2777-0/minicluster-data/master-0/wal/instance
I20260928 21:58:03.909425 2777 external_mini_cluster.cc:1428] Setting key f601d6dd822fcc642fc1c5c9f2eb385a
I20260928 21:58:03.911530 21425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66db1dbf2cdb405b85b7281ea8d73220: Bootstrap starting.
I20260928 21:58:03.912097 21425 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66db1dbf2cdb405b85b7281ea8d73220: Neither blocks nor log segments found. Creating new log.
I20260928 21:58:03.912340 21425 log.cc:824] T 00000000000000000000000000000000 P 66db1dbf2cdb405b85b7281ea8d73220: Log is configured to *not* fsync() on all Append() calls
I20260928 21:58:03.912964 21425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66db1dbf2cdb405b85b7281ea8d73220: No bootstrap required, opened a new log
I20260928 21:58:03.915030 21428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:03.910194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55782 (local address 127.2.182.126:39553)
0928 21:58:03.910420 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:03.910427 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:03.910452 (+ 25us) server_negotiation.cc:407] Connection header received
0928 21:58:03.910867 (+ 415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:03.910873 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:03.910933 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:03.911083 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:03.911284 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:03.912294 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:03.913092 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:03.913512 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:03.913597 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:03.914332 (+ 735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:03.914336 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:03.914336 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:58:03.914565 (+ 229us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:03.914598 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:58:03.914610 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:03.914657 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:03.914766 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 21:58:03.914828 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":91,"threads_started":1}
I20260928 21:58:03.915015 21425 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66db1dbf2cdb405b85b7281ea8d73220 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66db1dbf2cdb405b85b7281ea8d73220" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 39553 } }
I20260928 21:58:03.915179 21425 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66db1dbf2cdb405b85b7281ea8d73220 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:58:03.915223 21425 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66db1dbf2cdb405b85b7281ea8d73220 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66db1dbf2cdb405b85b7281ea8d73220, State: Initialized, Role: FOLLOWER
I20260928 21:58:03.915364 21425 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66db1dbf2cdb405b85b7281ea8d73220 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66db1dbf2cdb405b85b7281ea8d73220" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 39553 } }
I20260928 21:58:03.915437 21425 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66db1dbf2cdb405b85b7281ea8d73220 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:58:03.915469 21425 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66db1dbf2cdb405b85b7281ea8d73220 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:58:03.915518 21425 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66db1dbf2cdb405b85b7281ea8d73220 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:58:03.916056 21425 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66db1dbf2cdb405b85b7281ea8d73220 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66db1dbf2cdb405b85b7281ea8d73220" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 39553 } }
I20260928 21:58:03.916152 21425 leader_election.cc:302] T 00000000000000000000000000000000 P 66db1dbf2cdb405b85b7281ea8d73220 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66db1dbf2cdb405b85b7281ea8d73220; no voters:
I20260928 21:58:03.916309 21425 leader_election.cc:288] T 00000000000000000000000000000000 P 66db1dbf2cdb405b85b7281ea8d73220 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:58:03.916352 21430 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66db1dbf2cdb405b85b7281ea8d73220 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:58:03.916483 21430 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66db1dbf2cdb405b85b7281ea8d73220 [term 1 LEADER]: Becoming Leader. State: Replica: 66db1dbf2cdb405b85b7281ea8d73220, State: Running, Role: LEADER
I20260928 21:58:03.916579 21425 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66db1dbf2cdb405b85b7281ea8d73220 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:58:03.916575 21430 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66db1dbf2cdb405b85b7281ea8d73220 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66db1dbf2cdb405b85b7281ea8d73220" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 39553 } }
I20260928 21:58:03.916931 21431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66db1dbf2cdb405b85b7281ea8d73220 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66db1dbf2cdb405b85b7281ea8d73220" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66db1dbf2cdb405b85b7281ea8d73220" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 39553 } } }
I20260928 21:58:03.916939 21432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66db1dbf2cdb405b85b7281ea8d73220 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66db1dbf2cdb405b85b7281ea8d73220. Latest consensus state: current_term: 1 leader_uuid: "66db1dbf2cdb405b85b7281ea8d73220" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66db1dbf2cdb405b85b7281ea8d73220" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 39553 } } }
I20260928 21:58:03.917125 21432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66db1dbf2cdb405b85b7281ea8d73220 [sys.catalog]: This master's current role is: LEADER
I20260928 21:58:03.917096 21431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66db1dbf2cdb405b85b7281ea8d73220 [sys.catalog]: This master's current role is: LEADER
W20260928 21:58:03.918176 21446 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 66db1dbf2cdb405b85b7281ea8d73220: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:58:03.918219 21446 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:58:03.918273 21439 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:58:03.918443 21439 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:58:03.919900 21439 catalog_manager.cc:1347] Generated new cluster ID: abe54128dc8f4942b9e26f9c026d4247
I20260928 21:58:03.919955 21439 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:58:03.928444 21439 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:58:03.929137 21439 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:58:03.938702 21439 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 66db1dbf2cdb405b85b7281ea8d73220: Generated new TSK 0
I20260928 21:58:03.938908 21439 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:58:03.948539 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632440055392-2777-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632440055392-2777-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632440055392-2777-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.65:0
--local_ip_for_outbound_sockets=127.2.182.65
--webserver_interface=127.2.182.65
--webserver_port=0
--tserver_master_addrs=127.2.182.126:39553
--builtin_ntp_servers=127.2.182.84:37207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 21:58:04.032017 21449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:58:04.032226 21449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:58:04.032248 21449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:58:04.033478 21449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:58:04.033517 21449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:58:04.033542 21449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.65
I20260928 21:58:04.035037 21449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:37207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632440055392-2777-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632440055392-2777-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.182.65
--webserver_port=0
--tserver_master_addrs=127.2.182.126:39553
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21449
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.65
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632440055392-2777-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:58:04.035260 21449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:58:04.035573 21449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:58:04.041543 21454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:04.041862 21457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:04.041973 21455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:04.042096 21449 server_base.cc:1034] running on GCE node
I20260928 21:58:04.042276 21449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:58:04.042505 21449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:58:04.043660 21449 hybrid_clock.cc:648] HybridClock initialized: now 1790632684043643 us; error 29 us; skew 500 ppm
I20260928 21:58:04.044930 21449 webserver.cc:460] Webserver started at http://127.2.182.65:43573/ using document root <none> and password file <none>
I20260928 21:58:04.045162 21449 fs_manager.cc:359] Metadata directory not provided
I20260928 21:58:04.045212 21449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:58:04.045337 21449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:58:04.046402 21449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632440055392-2777-0/minicluster-data/ts-0/data/instance:
uuid: "a3b6bc8481f9444cbf16dc154793c681"
format_stamp: "Formatted at 2026-09-28 21:58:04 on dist-test-slave-k6lz"
server_key: "b0f3aafc9a72472088052dd1bafb6cba"
server_key_iv: "59cbe84c8bd7469d80ec09095aef264c"
server_key_version: "encryptionkey@0"
I20260928 21:58:04.046742 21449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632440055392-2777-0/minicluster-data/ts-0/wal/instance:
uuid: "a3b6bc8481f9444cbf16dc154793c681"
format_stamp: "Formatted at 2026-09-28 21:58:04 on dist-test-slave-k6lz"
server_key: "b0f3aafc9a72472088052dd1bafb6cba"
server_key_iv: "59cbe84c8bd7469d80ec09095aef264c"
server_key_version: "encryptionkey@0"
I20260928 21:58:04.048131 21449 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:58:04.048964 21463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:04.049160 21449 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:58:04.049255 21449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632440055392-2777-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632440055392-2777-0/minicluster-data/ts-0/wal
uuid: "a3b6bc8481f9444cbf16dc154793c681"
format_stamp: "Formatted at 2026-09-28 21:58:04 on dist-test-slave-k6lz"
server_key: "b0f3aafc9a72472088052dd1bafb6cba"
server_key_iv: "59cbe84c8bd7469d80ec09095aef264c"
server_key_version: "encryptionkey@0"
I20260928 21:58:04.049322 21449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632440055392-2777-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632440055392-2777-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632440055392-2777-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:58:04.069327 21449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:58:04.069666 21449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:58:04.069793 21449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:58:04.070007 21449 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:58:04.070310 21449 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:58:04.070341 21449 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:04.070370 21449 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:58:04.070393 21449 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:04.075636 21449 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.65:43315
I20260928 21:58:04.075707 21576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.65:43315 every 8 connection(s)
I20260928 21:58:04.076001 21449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
I20260928 21:58:04.081035 21428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:04.076689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.65:52941 (local address 127.2.182.126:39553)
0928 21:58:04.076750 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:04.076752 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:04.076987 (+ 235us) server_negotiation.cc:407] Connection header received
0928 21:58:04.077480 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:04.077481 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:04.077490 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:04.077551 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:04.078230 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:04.078792 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:04.079756 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:04.079959 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:04.080024 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:04.080795 (+ 771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:04.080796 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:04.080797 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:58:04.080801 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:04.080815 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:58:04.080818 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:04.080867 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:04.080943 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 21:58:04.080970 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 21:58:04.081218 21579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:04.076667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:39553 (local address 127.2.182.65:52941)
0928 21:58:04.076862 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:04.076901 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:04.077365 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:04.077663 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:04.077667 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:04.077675 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:58:04.078152 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:04.078156 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:04.078877 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:04.078878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:04.079676 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:04.079679 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:04.080044 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:04.080045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:04.080166 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:04.080667 (+ 501us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:58:04.080667 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:58:04.080694 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:58:04.080717 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:58:04.080883 (+ 166us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:58:04.080887 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:58:04.080889 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 21:58:04.080939 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 21:58:04.081018 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":85,"threads_started":1}
I20260928 21:58:04.081691 21577 heartbeater.cc:347] Connected to a master server at 127.2.182.126:39553
I20260928 21:58:04.081772 21577 heartbeater.cc:464] Registering TS with master...
I20260928 21:58:04.081935 21577 heartbeater.cc:511] Master 127.2.182.126:39553 requested a full tablet report, sending...
I20260928 21:58:04.082360 21389 ts_manager.cc:194] Registered new tserver with Master: a3b6bc8481f9444cbf16dc154793c681 (127.2.182.65:43315)
I20260928 21:58:04.083009 21389 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.182.65:52941
I20260928 21:58:04.083782 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 21449
I20260928 21:58:04.083870 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632440055392-2777-0/minicluster-data/ts-0/wal/instance
I20260928 21:58:04.084028 2777 external_mini_cluster.cc:1428] Setting key 9ad980d6b0586d0aa22f07fb90d14690
I20260928 21:58:04.084867 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632440055392-2777-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632440055392-2777-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632440055392-2777-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.66:0
--local_ip_for_outbound_sockets=127.2.182.66
--webserver_interface=127.2.182.66
--webserver_port=0
--tserver_master_addrs=127.2.182.126:39553
--builtin_ntp_servers=127.2.182.84:37207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 21:58:04.164229 21580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:58:04.164440 21580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:58:04.164461 21580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:58:04.165555 21580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:58:04.165586 21580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:58:04.165609 21580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.66
I20260928 21:58:04.166874 21580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:37207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632440055392-2777-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632440055392-2777-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.182.66
--webserver_port=0
--tserver_master_addrs=127.2.182.126:39553
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.66
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632440055392-2777-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:58:04.167052 21580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:58:04.167286 21580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:58:04.172123 21586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:04.172137 21585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:04.172138 21588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:04.172389 21580 server_base.cc:1034] running on GCE node
I20260928 21:58:04.172583 21580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:58:04.172802 21580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:58:04.173969 21580 hybrid_clock.cc:648] HybridClock initialized: now 1790632684173950 us; error 35 us; skew 500 ppm
I20260928 21:58:04.175304 21580 webserver.cc:460] Webserver started at http://127.2.182.66:45453/ using document root <none> and password file <none>
I20260928 21:58:04.175556 21580 fs_manager.cc:359] Metadata directory not provided
I20260928 21:58:04.175606 21580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:58:04.175748 21580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:58:04.176904 21580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632440055392-2777-0/minicluster-data/ts-1/data/instance:
uuid: "034c007fbdba4b78a7eba6c2645eda76"
format_stamp: "Formatted at 2026-09-28 21:58:04 on dist-test-slave-k6lz"
server_key: "7b10976cf1dd4184240ed8a8ba7c6351"
server_key_iv: "d46a075797aa1d95c3591734571d5a4a"
server_key_version: "encryptionkey@0"
I20260928 21:58:04.177230 21580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632440055392-2777-0/minicluster-data/ts-1/wal/instance:
uuid: "034c007fbdba4b78a7eba6c2645eda76"
format_stamp: "Formatted at 2026-09-28 21:58:04 on dist-test-slave-k6lz"
server_key: "7b10976cf1dd4184240ed8a8ba7c6351"
server_key_iv: "d46a075797aa1d95c3591734571d5a4a"
server_key_version: "encryptionkey@0"
I20260928 21:58:04.178682 21580 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:58:04.179584 21594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:04.179731 21580 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:58:04.179814 21580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632440055392-2777-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632440055392-2777-0/minicluster-data/ts-1/wal
uuid: "034c007fbdba4b78a7eba6c2645eda76"
format_stamp: "Formatted at 2026-09-28 21:58:04 on dist-test-slave-k6lz"
server_key: "7b10976cf1dd4184240ed8a8ba7c6351"
server_key_iv: "d46a075797aa1d95c3591734571d5a4a"
server_key_version: "encryptionkey@0"
I20260928 21:58:04.179878 21580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632440055392-2777-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632440055392-2777-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632440055392-2777-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:58:04.195118 21580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:58:04.195466 21580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:58:04.195598 21580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:58:04.195834 21580 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:58:04.196180 21580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:58:04.196214 21580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:04.196244 21580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:58:04.196265 21580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:04.201721 21580 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.66:44317
I20260928 21:58:04.201773 21707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.66:44317 every 8 connection(s)
I20260928 21:58:04.202090 21580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
I20260928 21:58:04.207625 21428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:04.202806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.66:45757 (local address 127.2.182.126:39553)
0928 21:58:04.202871 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:04.202872 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:04.203163 (+ 291us) server_negotiation.cc:407] Connection header received
0928 21:58:04.203800 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:04.203801 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:04.203813 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:04.203896 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:04.204651 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:04.205148 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:04.206130 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:04.206324 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:04.206380 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:04.207216 (+ 836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:04.207217 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:04.207217 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:58:04.207222 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:04.207235 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:58:04.207239 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:04.207282 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:04.207468 (+ 186us) server_negotiation.cc:299] Negotiation successful
0928 21:58:04.207498 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 21:58:04.207748 21710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:04.202782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:39553 (local address 127.2.182.66:45757)
0928 21:58:04.202985 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:04.203029 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:04.203669 (+ 640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:04.204001 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:04.204005 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:04.204014 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:58:04.204522 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:04.204528 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:04.205238 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:04.205239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:04.206044 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:04.206046 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:04.206418 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:04.206419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:04.206548 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:04.207076 (+ 528us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:58:04.207076 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:58:04.207105 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:58:04.207128 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:58:04.207313 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:58:04.207319 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:58:04.207322 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:58:04.207435 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 21:58:04.207530 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":85,"threads_started":1}
I20260928 21:58:04.208259 21708 heartbeater.cc:347] Connected to a master server at 127.2.182.126:39553
I20260928 21:58:04.208346 21708 heartbeater.cc:464] Registering TS with master...
I20260928 21:58:04.208520 21708 heartbeater.cc:511] Master 127.2.182.126:39553 requested a full tablet report, sending...
I20260928 21:58:04.208901 21389 ts_manager.cc:194] Registered new tserver with Master: 034c007fbdba4b78a7eba6c2645eda76 (127.2.182.66:44317)
I20260928 21:58:04.209316 21389 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.182.66:45757
I20260928 21:58:04.209785 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 21580
I20260928 21:58:04.209883 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632440055392-2777-0/minicluster-data/ts-1/wal/instance
I20260928 21:58:04.210116 2777 external_mini_cluster.cc:1428] Setting key 513abd46dbf76bae0e24f2829056497b
I20260928 21:58:04.211018 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632440055392-2777-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632440055392-2777-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632440055392-2777-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.67:0
--local_ip_for_outbound_sockets=127.2.182.67
--webserver_interface=127.2.182.67
--webserver_port=0
--tserver_master_addrs=127.2.182.126:39553
--builtin_ntp_servers=127.2.182.84:37207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 21:58:04.290699 21711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:58:04.290904 21711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:58:04.290933 21711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:58:04.292109 21711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:58:04.292148 21711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:58:04.292181 21711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.67
I20260928 21:58:04.293504 21711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:37207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632440055392-2777-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632440055392-2777-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.182.67
--webserver_port=0
--tserver_master_addrs=127.2.182.126:39553
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.67
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632440055392-2777-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:58:04.293747 21711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:58:04.294029 21711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:58:04.299162 21716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:04.299167 21717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:04.299157 21719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:04.299672 21711 server_base.cc:1034] running on GCE node
I20260928 21:58:04.300112 21711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:58:04.300460 21711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:58:04.301642 21711 hybrid_clock.cc:648] HybridClock initialized: now 1790632684301623 us; error 37 us; skew 500 ppm
I20260928 21:58:04.302992 21711 webserver.cc:460] Webserver started at http://127.2.182.67:33111/ using document root <none> and password file <none>
I20260928 21:58:04.303210 21711 fs_manager.cc:359] Metadata directory not provided
I20260928 21:58:04.303259 21711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:58:04.303470 21711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:58:04.304574 21711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632440055392-2777-0/minicluster-data/ts-2/data/instance:
uuid: "ea0cd8ab5ba84af0a17b71525ae3740e"
format_stamp: "Formatted at 2026-09-28 21:58:04 on dist-test-slave-k6lz"
server_key: "8295c91ef62c1c89c72fc870d7530600"
server_key_iv: "ccb7bcee70a430905261d68168a80962"
server_key_version: "encryptionkey@0"
I20260928 21:58:04.304910 21711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632440055392-2777-0/minicluster-data/ts-2/wal/instance:
uuid: "ea0cd8ab5ba84af0a17b71525ae3740e"
format_stamp: "Formatted at 2026-09-28 21:58:04 on dist-test-slave-k6lz"
server_key: "8295c91ef62c1c89c72fc870d7530600"
server_key_iv: "ccb7bcee70a430905261d68168a80962"
server_key_version: "encryptionkey@0"
I20260928 21:58:04.306236 21711 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 21:58:04.307089 21725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:04.307271 21711 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:58:04.307374 21711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632440055392-2777-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632440055392-2777-0/minicluster-data/ts-2/wal
uuid: "ea0cd8ab5ba84af0a17b71525ae3740e"
format_stamp: "Formatted at 2026-09-28 21:58:04 on dist-test-slave-k6lz"
server_key: "8295c91ef62c1c89c72fc870d7530600"
server_key_iv: "ccb7bcee70a430905261d68168a80962"
server_key_version: "encryptionkey@0"
I20260928 21:58:04.307444 21711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632440055392-2777-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632440055392-2777-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632440055392-2777-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:58:04.318964 21711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:58:04.319262 21711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:58:04.319416 21711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:58:04.319638 21711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:58:04.319967 21711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:58:04.320004 21711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:04.320036 21711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:58:04.320052 21711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:04.325487 21711 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.67:35089
I20260928 21:58:04.325560 21838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.67:35089 every 8 connection(s)
I20260928 21:58:04.325868 21711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
I20260928 21:58:04.331358 21428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:04.326685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.67:50639 (local address 127.2.182.126:39553)
0928 21:58:04.326763 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:04.326765 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:04.327025 (+ 260us) server_negotiation.cc:407] Connection header received
0928 21:58:04.327612 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:04.327613 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:04.327624 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:04.327680 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:04.328398 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:04.328918 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:04.329959 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:04.330156 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:04.330218 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:04.331076 (+ 858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:04.331078 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:04.331078 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:58:04.331083 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:04.331098 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:58:04.331101 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:04.331151 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:04.331235 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 21:58:04.331259 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 21:58:04.331600 21841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:04.326672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:39553 (local address 127.2.182.67:50639)
0928 21:58:04.326886 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:04.326923 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:04.327444 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:04.327750 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:04.327754 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:04.327761 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:58:04.328293 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:04.328297 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:04.329016 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:04.329017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:04.329851 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:04.329860 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:04.330269 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:04.330270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:04.330391 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:04.330927 (+ 536us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:58:04.330928 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:58:04.330956 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:58:04.330979 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:58:04.331167 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:58:04.331172 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:58:04.331175 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:58:04.331228 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 21:58:04.331312 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":84,"threads_started":1}
I20260928 21:58:04.332080 21839 heartbeater.cc:347] Connected to a master server at 127.2.182.126:39553
I20260928 21:58:04.332160 21839 heartbeater.cc:464] Registering TS with master...
I20260928 21:58:04.332322 21839 heartbeater.cc:511] Master 127.2.182.126:39553 requested a full tablet report, sending...
I20260928 21:58:04.332710 21389 ts_manager.cc:194] Registered new tserver with Master: ea0cd8ab5ba84af0a17b71525ae3740e (127.2.182.67:35089)
I20260928 21:58:04.333201 21389 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.182.67:50639
I20260928 21:58:04.335777 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 21711
I20260928 21:58:04.335860 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632440055392-2777-0/minicluster-data/ts-2/wal/instance
I20260928 21:58:04.336058 2777 external_mini_cluster.cc:1428] Setting key a8bfe334dc0636a3ed05e25afd792c2a
I20260928 21:58:04.337097 2777 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:58:04.341538 21428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:04.338580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55788 (local address 127.2.182.126:39553)
0928 21:58:04.338664 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:04.338666 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:04.338735 (+ 69us) server_negotiation.cc:407] Connection header received
0928 21:58:04.338795 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:04.338796 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:04.338811 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:04.338858 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:04.339013 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:04.339536 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:04.340289 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:04.340526 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:04.340583 (+ 57us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:04.341238 (+ 655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:04.341239 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:04.341240 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:58:04.341245 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:04.341258 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:58:04.341262 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:04.341336 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:04.341448 (+ 112us) server_negotiation.cc:299] Negotiation successful
0928 21:58:04.341458 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 21:58:04.343055 21389 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55788:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:58:04.344607 21389 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:58:04.350636 21849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:04.346654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38570 (local address 127.2.182.65:43315)
0928 21:58:04.346819 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:04.346824 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:04.346837 (+ 13us) server_negotiation.cc:407] Connection header received
0928 21:58:04.346980 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:04.346982 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:04.346997 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:04.347044 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:58:04.347229 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:04.348380 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:04.349740 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:04.350239 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:04.350309 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:04.350394 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:04.350506 (+ 112us) server_negotiation.cc:299] Negotiation successful
0928 21:58:04.350545 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":94,"threads_started":1}
I20260928 21:58:04.350653 21848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:04.346531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.65:43315 (local address 127.0.0.1:38570)
0928 21:58:04.346751 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:04.346793 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:04.346857 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:04.347112 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:04.347114 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:04.347115 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:58:04.347165 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:04.347166 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:04.348469 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:04.348470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:04.349598 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:04.349599 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:04.350319 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:04.350320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:04.350397 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:04.350410 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 21:58:04.350477 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 21:58:04.350543 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":11,"thread_start_us":69,"threads_started":1}
I20260928 21:58:04.350829 21851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:04.346910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44044 (local address 127.2.182.66:44317)
0928 21:58:04.347128 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:04.347132 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:04.347246 (+ 114us) server_negotiation.cc:407] Connection header received
0928 21:58:04.347295 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:04.347298 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:04.347313 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:04.347396 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:58:04.347566 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:04.348642 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:04.349742 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:04.350236 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:04.350294 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:04.350385 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:04.350698 (+ 313us) server_negotiation.cc:299] Negotiation successful
0928 21:58:04.350749 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":75,"threads_started":1}
I20260928 21:58:04.350847 21847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:04.346731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.66:44317 (local address 127.0.0.1:44044)
0928 21:58:04.347181 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:04.347190 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:04.347241 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:04.347462 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:04.347463 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:04.347465 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:58:04.347499 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:04.347500 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:04.348716 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:04.348717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:04.349645 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:04.349656 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:04.350322 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:04.350323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:04.350400 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:04.350410 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 21:58:04.350550 (+ 140us) client_negotiation.cc:240] Negotiation successful
0928 21:58:04.350593 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"mutex_wait_us":1,"thread_start_us":59,"threads_started":1}
I20260928 21:58:04.351622 21850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:04.346608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.67:35089 (local address 127.0.0.1:57076)
0928 21:58:04.346882 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:04.346891 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:04.346928 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:04.347287 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:04.347288 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:04.347290 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:58:04.347357 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:04.347359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:04.348269 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:04.348270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:04.349595 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:04.349598 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:04.351404 (+ 1806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:04.351405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:04.351463 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:04.351466 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:58:04.351513 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 21:58:04.351557 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":80,"threads_started":1}
I20260928 21:58:04.351671 21852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:04.346909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57076 (local address 127.2.182.67:35089)
0928 21:58:04.347076 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:04.347080 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:04.347091 (+ 11us) server_negotiation.cc:407] Connection header received
0928 21:58:04.347129 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:04.347131 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:04.347146 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:04.347227 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:58:04.347429 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:04.348190 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:04.350445 (+ 2255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:04.351175 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:04.351240 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:04.351322 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:04.351540 (+ 218us) server_negotiation.cc:299] Negotiation successful
0928 21:58:04.351598 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":77,"threads_started":1}
I20260928 21:58:04.351666 21642 tablet_service.cc:1467] Processing CreateTablet for tablet 7896c4c50c0f45499fac5907fff942b2 (DEFAULT_TABLE table=test-table [id=04c4551caa4644489894c3925ef060c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:58:04.352034 21642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7896c4c50c0f45499fac5907fff942b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:58:04.354758 21853 tablet_bootstrap.cc:492] T 7896c4c50c0f45499fac5907fff942b2 P 034c007fbdba4b78a7eba6c2645eda76: Bootstrap starting.
I20260928 21:58:04.355036 21511 tablet_service.cc:1467] Processing CreateTablet for tablet 7896c4c50c0f45499fac5907fff942b2 (DEFAULT_TABLE table=test-table [id=04c4551caa4644489894c3925ef060c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:58:04.355424 21511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7896c4c50c0f45499fac5907fff942b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:58:04.355480 21853 tablet_bootstrap.cc:654] T 7896c4c50c0f45499fac5907fff942b2 P 034c007fbdba4b78a7eba6c2645eda76: Neither blocks nor log segments found. Creating new log.
I20260928 21:58:04.355762 21853 log.cc:824] T 7896c4c50c0f45499fac5907fff942b2 P 034c007fbdba4b78a7eba6c2645eda76: Log is configured to *not* fsync() on all Append() calls
I20260928 21:58:04.356707 21853 tablet_bootstrap.cc:492] T 7896c4c50c0f45499fac5907fff942b2 P 034c007fbdba4b78a7eba6c2645eda76: No bootstrap required, opened a new log
I20260928 21:58:04.356783 21853 ts_tablet_manager.cc:1397] T 7896c4c50c0f45499fac5907fff942b2 P 034c007fbdba4b78a7eba6c2645eda76: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 21:58:04.358219 21855 tablet_bootstrap.cc:492] T 7896c4c50c0f45499fac5907fff942b2 P a3b6bc8481f9444cbf16dc154793c681: Bootstrap starting.
I20260928 21:58:04.358862 21853 raft_consensus.cc:348] T 7896c4c50c0f45499fac5907fff942b2 P 034c007fbdba4b78a7eba6c2645eda76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3b6bc8481f9444cbf16dc154793c681" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 43315 } } peers { permanent_uuid: "ea0cd8ab5ba84af0a17b71525ae3740e" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 35089 } } peers { permanent_uuid: "034c007fbdba4b78a7eba6c2645eda76" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 44317 } }
I20260928 21:58:04.358876 21773 tablet_service.cc:1467] Processing CreateTablet for tablet 7896c4c50c0f45499fac5907fff942b2 (DEFAULT_TABLE table=test-table [id=04c4551caa4644489894c3925ef060c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:58:04.359004 21853 raft_consensus.cc:374] T 7896c4c50c0f45499fac5907fff942b2 P 034c007fbdba4b78a7eba6c2645eda76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:58:04.359037 21853 raft_consensus.cc:729] T 7896c4c50c0f45499fac5907fff942b2 P 034c007fbdba4b78a7eba6c2645eda76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 034c007fbdba4b78a7eba6c2645eda76, State: Initialized, Role: FOLLOWER
I20260928 21:58:04.359050 21855 tablet_bootstrap.cc:654] T 7896c4c50c0f45499fac5907fff942b2 P a3b6bc8481f9444cbf16dc154793c681: Neither blocks nor log segments found. Creating new log.
I20260928 21:58:04.359153 21853 consensus_queue.cc:260] T 7896c4c50c0f45499fac5907fff942b2 P 034c007fbdba4b78a7eba6c2645eda76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3b6bc8481f9444cbf16dc154793c681" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 43315 } } peers { permanent_uuid: "ea0cd8ab5ba84af0a17b71525ae3740e" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 35089 } } peers { permanent_uuid: "034c007fbdba4b78a7eba6c2645eda76" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 44317 } }
I20260928 21:58:04.359211 21773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7896c4c50c0f45499fac5907fff942b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:58:04.359449 21853 ts_tablet_manager.cc:1428] T 7896c4c50c0f45499fac5907fff942b2 P 034c007fbdba4b78a7eba6c2645eda76: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 21:58:04.359485 21708 heartbeater.cc:503] Master 127.2.182.126:39553 was elected leader, sending a full tablet report...
I20260928 21:58:04.359413 21855 log.cc:824] T 7896c4c50c0f45499fac5907fff942b2 P a3b6bc8481f9444cbf16dc154793c681: Log is configured to *not* fsync() on all Append() calls
I20260928 21:58:04.360477 21855 tablet_bootstrap.cc:492] T 7896c4c50c0f45499fac5907fff942b2 P a3b6bc8481f9444cbf16dc154793c681: No bootstrap required, opened a new log
I20260928 21:58:04.360545 21855 ts_tablet_manager.cc:1397] T 7896c4c50c0f45499fac5907fff942b2 P a3b6bc8481f9444cbf16dc154793c681: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 21:58:04.362430 21858 tablet_bootstrap.cc:492] T 7896c4c50c0f45499fac5907fff942b2 P ea0cd8ab5ba84af0a17b71525ae3740e: Bootstrap starting.
I20260928 21:58:04.362989 21855 raft_consensus.cc:348] T 7896c4c50c0f45499fac5907fff942b2 P a3b6bc8481f9444cbf16dc154793c681 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3b6bc8481f9444cbf16dc154793c681" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 43315 } } peers { permanent_uuid: "ea0cd8ab5ba84af0a17b71525ae3740e" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 35089 } } peers { permanent_uuid: "034c007fbdba4b78a7eba6c2645eda76" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 44317 } }
I20260928 21:58:04.363123 21858 tablet_bootstrap.cc:654] T 7896c4c50c0f45499fac5907fff942b2 P ea0cd8ab5ba84af0a17b71525ae3740e: Neither blocks nor log segments found. Creating new log.
I20260928 21:58:04.363157 21855 raft_consensus.cc:374] T 7896c4c50c0f45499fac5907fff942b2 P a3b6bc8481f9444cbf16dc154793c681 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:58:04.363205 21855 raft_consensus.cc:729] T 7896c4c50c0f45499fac5907fff942b2 P a3b6bc8481f9444cbf16dc154793c681 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3b6bc8481f9444cbf16dc154793c681, State: Initialized, Role: FOLLOWER
I20260928 21:58:04.363310 21855 consensus_queue.cc:260] T 7896c4c50c0f45499fac5907fff942b2 P a3b6bc8481f9444cbf16dc154793c681 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3b6bc8481f9444cbf16dc154793c681" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 43315 } } peers { permanent_uuid: "ea0cd8ab5ba84af0a17b71525ae3740e" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 35089 } } peers { permanent_uuid: "034c007fbdba4b78a7eba6c2645eda76" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 44317 } }
I20260928 21:58:04.363437 21858 log.cc:824] T 7896c4c50c0f45499fac5907fff942b2 P ea0cd8ab5ba84af0a17b71525ae3740e: Log is configured to *not* fsync() on all Append() calls
I20260928 21:58:04.363584 21855 ts_tablet_manager.cc:1428] T 7896c4c50c0f45499fac5907fff942b2 P a3b6bc8481f9444cbf16dc154793c681: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260928 21:58:04.363620 21577 heartbeater.cc:503] Master 127.2.182.126:39553 was elected leader, sending a full tablet report...
I20260928 21:58:04.364146 21858 tablet_bootstrap.cc:492] T 7896c4c50c0f45499fac5907fff942b2 P ea0cd8ab5ba84af0a17b71525ae3740e: No bootstrap required, opened a new log
I20260928 21:58:04.364197 21858 ts_tablet_manager.cc:1397] T 7896c4c50c0f45499fac5907fff942b2 P ea0cd8ab5ba84af0a17b71525ae3740e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:58:04.365921 21858 raft_consensus.cc:348] T 7896c4c50c0f45499fac5907fff942b2 P ea0cd8ab5ba84af0a17b71525ae3740e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3b6bc8481f9444cbf16dc154793c681" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 43315 } } peers { permanent_uuid: "ea0cd8ab5ba84af0a17b71525ae3740e" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 35089 } } peers { permanent_uuid: "034c007fbdba4b78a7eba6c2645eda76" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 44317 } }
I20260928 21:58:04.366045 21858 raft_consensus.cc:374] T 7896c4c50c0f45499fac5907fff942b2 P ea0cd8ab5ba84af0a17b71525ae3740e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:58:04.366066 21858 raft_consensus.cc:729] T 7896c4c50c0f45499fac5907fff942b2 P ea0cd8ab5ba84af0a17b71525ae3740e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea0cd8ab5ba84af0a17b71525ae3740e, State: Initialized, Role: FOLLOWER
I20260928 21:58:04.366169 21858 consensus_queue.cc:260] T 7896c4c50c0f45499fac5907fff942b2 P ea0cd8ab5ba84af0a17b71525ae3740e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3b6bc8481f9444cbf16dc154793c681" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 43315 } } peers { permanent_uuid: "ea0cd8ab5ba84af0a17b71525ae3740e" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 35089 } } peers { permanent_uuid: "034c007fbdba4b78a7eba6c2645eda76" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 44317 } }
I20260928 21:58:04.366379 21858 ts_tablet_manager.cc:1428] T 7896c4c50c0f45499fac5907fff942b2 P ea0cd8ab5ba84af0a17b71525ae3740e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 21:58:04.366499 21839 heartbeater.cc:503] Master 127.2.182.126:39553 was elected leader, sending a full tablet report...
W20260928 21:58:04.452700 21709 tablet.cc:2369] T 7896c4c50c0f45499fac5907fff942b2 P 034c007fbdba4b78a7eba6c2645eda76: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:58:04.509088 21856 raft_consensus.cc:482] T 7896c4c50c0f45499fac5907fff942b2 P 034c007fbdba4b78a7eba6c2645eda76 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:58:04.509238 21856 raft_consensus.cc:504] T 7896c4c50c0f45499fac5907fff942b2 P 034c007fbdba4b78a7eba6c2645eda76 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3b6bc8481f9444cbf16dc154793c681" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 43315 } } peers { permanent_uuid: "ea0cd8ab5ba84af0a17b71525ae3740e" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 35089 } } peers { permanent_uuid: "034c007fbdba4b78a7eba6c2645eda76" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 44317 } }
I20260928 21:58:04.509572 21856 leader_election.cc:288] T 7896c4c50c0f45499fac5907fff942b2 P 034c007fbdba4b78a7eba6c2645eda76 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3b6bc8481f9444cbf16dc154793c681 (127.2.182.65:43315), ea0cd8ab5ba84af0a17b71525ae3740e (127.2.182.67:35089)
I20260928 21:58:04.512444 21710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:04.509743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.67:35089 (local address 127.2.182.66:52901)
0928 21:58:04.509788 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:04.509800 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:04.509847 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:04.509967 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:04.509968 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:04.509969 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:58:04.510063 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:04.510066 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:04.510782 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:04.510783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:04.511695 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:04.511698 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:04.512195 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:04.512196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:04.512265 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:04.512268 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:58:04.512311 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 21:58:04.512354 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24,"mutex_wait_us":30}
I20260928 21:58:04.512428 21852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:04.509820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.66:52901 (local address 127.2.182.67:35089)
0928 21:58:04.509863 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:04.509865 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:04.509875 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:58:04.509904 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:04.509904 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:04.509914 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:04.509955 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:58:04.510118 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:04.510704 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:04.511777 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:04.512130 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:04.512190 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:04.512199 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:04.512322 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 21:58:04.512357 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 21:58:04.512442 21849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:04.509862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.66:58215 (local address 127.2.182.65:43315)
0928 21:58:04.509906 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:04.509908 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:04.510011 (+ 103us) server_negotiation.cc:407] Connection header received
0928 21:58:04.510040 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:04.510041 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:04.510051 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:04.510095 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:58:04.510230 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:04.510700 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:04.511779 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:04.512139 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:04.512198 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:04.512206 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:04.512338 (+ 132us) server_negotiation.cc:299] Negotiation successful
0928 21:58:04.512378 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 21:58:04.512459 21862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:04.509747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.65:43315 (local address 127.2.182.66:58215)
0928 21:58:04.509918 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:04.509928 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:04.509985 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:04.510105 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:04.510106 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:04.510108 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:58:04.510173 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:04.510175 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:04.510768 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:04.510769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:04.511697 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:04.511700 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:04.512223 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:04.512224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:04.512279 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:04.512282 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:58:04.512326 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 21:58:04.512362 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":14,"thread_start_us":89,"threads_started":1}
I20260928 21:58:04.512787 21793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7896c4c50c0f45499fac5907fff942b2" candidate_uuid: "034c007fbdba4b78a7eba6c2645eda76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea0cd8ab5ba84af0a17b71525ae3740e" is_pre_election: true
I20260928 21:58:04.512948 21793 raft_consensus.cc:2454] T 7896c4c50c0f45499fac5907fff942b2 P ea0cd8ab5ba84af0a17b71525ae3740e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 034c007fbdba4b78a7eba6c2645eda76 in term 0.
I20260928 21:58:04.512938 21531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7896c4c50c0f45499fac5907fff942b2" candidate_uuid: "034c007fbdba4b78a7eba6c2645eda76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3b6bc8481f9444cbf16dc154793c681" is_pre_election: true
I20260928 21:58:04.513068 21531 raft_consensus.cc:2454] T 7896c4c50c0f45499fac5907fff942b2 P a3b6bc8481f9444cbf16dc154793c681 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 034c007fbdba4b78a7eba6c2645eda76 in term 0.
I20260928 21:58:04.513171 21596 leader_election.cc:302] T 7896c4c50c0f45499fac5907fff942b2 P 034c007fbdba4b78a7eba6c2645eda76 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 034c007fbdba4b78a7eba6c2645eda76, ea0cd8ab5ba84af0a17b71525ae3740e; no voters:
I20260928 21:58:04.513291 21856 raft_consensus.cc:2781] T 7896c4c50c0f45499fac5907fff942b2 P 034c007fbdba4b78a7eba6c2645eda76 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:58:04.513350 21856 raft_consensus.cc:482] T 7896c4c50c0f45499fac5907fff942b2 P 034c007fbdba4b78a7eba6c2645eda76 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:58:04.513367 21856 raft_consensus.cc:3037] T 7896c4c50c0f45499fac5907fff942b2 P 034c007fbdba4b78a7eba6c2645eda76 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:58:04.514027 21856 raft_consensus.cc:504] T 7896c4c50c0f45499fac5907fff942b2 P 034c007fbdba4b78a7eba6c2645eda76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3b6bc8481f9444cbf16dc154793c681" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 43315 } } peers { permanent_uuid: "ea0cd8ab5ba84af0a17b71525ae3740e" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 35089 } } peers { permanent_uuid: "034c007fbdba4b78a7eba6c2645eda76" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 44317 } }
I20260928 21:58:04.514142 21856 leader_election.cc:288] T 7896c4c50c0f45499fac5907fff942b2 P 034c007fbdba4b78a7eba6c2645eda76 [CANDIDATE]: Term 1 election: Requested vote from peers a3b6bc8481f9444cbf16dc154793c681 (127.2.182.65:43315), ea0cd8ab5ba84af0a17b71525ae3740e (127.2.182.67:35089)
I20260928 21:58:04.514304 21793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7896c4c50c0f45499fac5907fff942b2" candidate_uuid: "034c007fbdba4b78a7eba6c2645eda76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea0cd8ab5ba84af0a17b71525ae3740e"
I20260928 21:58:04.514297 21531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7896c4c50c0f45499fac5907fff942b2" candidate_uuid: "034c007fbdba4b78a7eba6c2645eda76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3b6bc8481f9444cbf16dc154793c681"
I20260928 21:58:04.514387 21793 raft_consensus.cc:3037] T 7896c4c50c0f45499fac5907fff942b2 P ea0cd8ab5ba84af0a17b71525ae3740e [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:58:04.514387 21531 raft_consensus.cc:3037] T 7896c4c50c0f45499fac5907fff942b2 P a3b6bc8481f9444cbf16dc154793c681 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:58:04.515300 21531 raft_consensus.cc:2454] T 7896c4c50c0f45499fac5907fff942b2 P a3b6bc8481f9444cbf16dc154793c681 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 034c007fbdba4b78a7eba6c2645eda76 in term 1.
I20260928 21:58:04.515300 21793 raft_consensus.cc:2454] T 7896c4c50c0f45499fac5907fff942b2 P ea0cd8ab5ba84af0a17b71525ae3740e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 034c007fbdba4b78a7eba6c2645eda76 in term 1.
I20260928 21:58:04.515504 21595 leader_election.cc:302] T 7896c4c50c0f45499fac5907fff942b2 P 034c007fbdba4b78a7eba6c2645eda76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 034c007fbdba4b78a7eba6c2645eda76, a3b6bc8481f9444cbf16dc154793c681; no voters:
I20260928 21:58:04.515623 21856 raft_consensus.cc:2781] T 7896c4c50c0f45499fac5907fff942b2 P 034c007fbdba4b78a7eba6c2645eda76 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:58:04.515785 21856 raft_consensus.cc:686] T 7896c4c50c0f45499fac5907fff942b2 P 034c007fbdba4b78a7eba6c2645eda76 [term 1 LEADER]: Becoming Leader. State: Replica: 034c007fbdba4b78a7eba6c2645eda76, State: Running, Role: LEADER
I20260928 21:58:04.515872 21856 consensus_queue.cc:237] T 7896c4c50c0f45499fac5907fff942b2 P 034c007fbdba4b78a7eba6c2645eda76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3b6bc8481f9444cbf16dc154793c681" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 43315 } } peers { permanent_uuid: "ea0cd8ab5ba84af0a17b71525ae3740e" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 35089 } } peers { permanent_uuid: "034c007fbdba4b78a7eba6c2645eda76" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 44317 } }
I20260928 21:58:04.516695 21389 catalog_manager.cc:5495] T 7896c4c50c0f45499fac5907fff942b2 P 034c007fbdba4b78a7eba6c2645eda76 reported cstate change: term changed from 0 to 1, leader changed from <none> to 034c007fbdba4b78a7eba6c2645eda76 (127.2.182.66). New cstate: current_term: 1 leader_uuid: "034c007fbdba4b78a7eba6c2645eda76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3b6bc8481f9444cbf16dc154793c681" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 43315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea0cd8ab5ba84af0a17b71525ae3740e" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 35089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "034c007fbdba4b78a7eba6c2645eda76" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 44317 } health_report { overall_health: HEALTHY } } }
I20260928 21:58:04.533195 21851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:04.530851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44054 (local address 127.2.182.66:44317)
0928 21:58:04.530912 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:04.530915 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:04.530923 (+ 8us) server_negotiation.cc:407] Connection header received
0928 21:58:04.530963 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:04.530964 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:04.530987 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:04.531044 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:58:04.531161 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:04.531698 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:04.532436 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:04.532646 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:04.532700 (+ 54us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:04.532843 (+ 143us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:58:04.532941 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:58:04.533034 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:04.533080 (+ 46us) server_negotiation.cc:299] Negotiation successful
0928 21:58:04.533088 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 21:58:04.535111 21793 raft_consensus.cc:1261] T 7896c4c50c0f45499fac5907fff942b2 P ea0cd8ab5ba84af0a17b71525ae3740e [term 1 FOLLOWER]: Refusing update from remote peer 034c007fbdba4b78a7eba6c2645eda76: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:58:04.535111 21531 raft_consensus.cc:1261] T 7896c4c50c0f45499fac5907fff942b2 P a3b6bc8481f9444cbf16dc154793c681 [term 1 FOLLOWER]: Refusing update from remote peer 034c007fbdba4b78a7eba6c2645eda76: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:58:04.535519 21856 consensus_queue.cc:1035] T 7896c4c50c0f45499fac5907fff942b2 P 034c007fbdba4b78a7eba6c2645eda76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea0cd8ab5ba84af0a17b71525ae3740e" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 35089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:58:04.535681 21863 consensus_queue.cc:1035] T 7896c4c50c0f45499fac5907fff942b2 P 034c007fbdba4b78a7eba6c2645eda76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3b6bc8481f9444cbf16dc154793c681" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 43315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:58:04.538563 21866 mvcc.cc:205] Tried to move back new op lower bound from 7334431475853074432 to 7334431475777634304. Current Snapshot: MvccSnapshot[applied={T|T < 7334431475853074432 or (T in {7334431475853074432})}]
I20260928 21:58:04.538708 21868 mvcc.cc:205] Tried to move back new op lower bound from 7334431475853074432 to 7334431475777634304. Current Snapshot: MvccSnapshot[applied={T|T < 7334431475853074432 or (T in {7334431475853074432})}]
I20260928 21:58:04.539490 21867 mvcc.cc:205] Tried to move back new op lower bound from 7334431475853074432 to 7334431475777634304. Current Snapshot: MvccSnapshot[applied={T|T < 7334431475853074432 or (T in {7334431475853074432})}]
I20260928 21:58:04.540268 21389 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55788:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:58:04.540349 21389 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55788:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:58:04.541082 21389 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 66db1dbf2cdb405b85b7281ea8d73220: Sending DeleteTablet for 3 replicas of tablet 7896c4c50c0f45499fac5907fff942b2
I20260928 21:58:04.541368 21642 tablet_service.cc:1514] Processing DeleteTablet for tablet 7896c4c50c0f45499fac5907fff942b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:58:04 UTC) from {username='slave'} at 127.0.0.1:44044
I20260928 21:58:04.541369 21773 tablet_service.cc:1514] Processing DeleteTablet for tablet 7896c4c50c0f45499fac5907fff942b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:58:04 UTC) from {username='slave'} at 127.0.0.1:57076
I20260928 21:58:04.541579 21875 tablet_replica.cc:330] T 7896c4c50c0f45499fac5907fff942b2 P ea0cd8ab5ba84af0a17b71525ae3740e: stopping tablet replica
I20260928 21:58:04.541586 21511 tablet_service.cc:1514] Processing DeleteTablet for tablet 7896c4c50c0f45499fac5907fff942b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:58:04 UTC) from {username='slave'} at 127.0.0.1:38570
I20260928 21:58:04.541675 21875 raft_consensus.cc:2229] T 7896c4c50c0f45499fac5907fff942b2 P ea0cd8ab5ba84af0a17b71525ae3740e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:58:04.541779 21875 raft_consensus.cc:2258] T 7896c4c50c0f45499fac5907fff942b2 P ea0cd8ab5ba84af0a17b71525ae3740e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:58:04.541796 21877 tablet_replica.cc:330] T 7896c4c50c0f45499fac5907fff942b2 P a3b6bc8481f9444cbf16dc154793c681: stopping tablet replica
I20260928 21:58:04.541960 21877 raft_consensus.cc:2229] T 7896c4c50c0f45499fac5907fff942b2 P a3b6bc8481f9444cbf16dc154793c681 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:58:04.542076 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 21449
I20260928 21:58:04.542088 21877 raft_consensus.cc:2258] T 7896c4c50c0f45499fac5907fff942b2 P a3b6bc8481f9444cbf16dc154793c681 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:58:04.542227 21875 ts_tablet_manager.cc:1905] T 7896c4c50c0f45499fac5907fff942b2 P ea0cd8ab5ba84af0a17b71525ae3740e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:58:04.544265 21875 ts_tablet_manager.cc:1918] T 7896c4c50c0f45499fac5907fff942b2 P ea0cd8ab5ba84af0a17b71525ae3740e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:58:04.544339 21875 log.cc:1196] T 7896c4c50c0f45499fac5907fff942b2 P ea0cd8ab5ba84af0a17b71525ae3740e: Deleting WAL directory at /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632440055392-2777-0/minicluster-data/ts-2/wal/wals/7896c4c50c0f45499fac5907fff942b2
I20260928 21:58:04.544586 21875 ts_tablet_manager.cc:1939] T 7896c4c50c0f45499fac5907fff942b2 P ea0cd8ab5ba84af0a17b71525ae3740e: Deleting consensus metadata
I20260928 21:58:04.544965 21374 catalog_manager.cc:4841] TS ea0cd8ab5ba84af0a17b71525ae3740e (127.2.182.67:35089): tablet 7896c4c50c0f45499fac5907fff942b2 (table test-table [id=04c4551caa4644489894c3925ef060c4]) successfully deleted
W20260928 21:58:04.547087 21373 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.182.65:43315 (error 108)
I20260928 21:58:04.547314 21876 tablet_replica.cc:330] T 7896c4c50c0f45499fac5907fff942b2 P 034c007fbdba4b78a7eba6c2645eda76: stopping tablet replica
I20260928 21:58:04.547441 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 21580
I20260928 21:58:04.547475 21876 raft_consensus.cc:2229] T 7896c4c50c0f45499fac5907fff942b2 P 034c007fbdba4b78a7eba6c2645eda76 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:58:04.547580 21876 raft_consensus.cc:2258] T 7896c4c50c0f45499fac5907fff942b2 P 034c007fbdba4b78a7eba6c2645eda76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:58:04.547978 21850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:04.547480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.65:43315 (local address 127.0.0.1:38582)
0928 21:58:04.547895 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:04.547946 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.65:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387}
W20260928 21:58:04.548074 21373 catalog_manager.cc:4584] TS a3b6bc8481f9444cbf16dc154793c681 (127.2.182.65:43315): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7896c4c50c0f45499fac5907fff942b2: Network error: Client connection negotiation failed: client connection to 127.2.182.65:43315: connect: Connection refused (error 111)
I20260928 21:58:04.551651 21850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:04.551560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.66:44317 (local address 127.0.0.1:44060)
0928 21:58:04.551601 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:04.551622 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.66:44317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260928 21:58:04.552250 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 21711
I20260928 21:58:04.556373 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 21358
2026-09-28T21:58:04Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (787 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:58:04 dist-test-slave-k6lz krb5kdc[21881](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:58:04 dist-test-slave-k6lz krb5kdc[21881](info): set up 2 sockets
Sep 28 21:58:04 dist-test-slave-k6lz krb5kdc[21881](info): commencing operation
krb5kdc: starting...
W20260928 21:58:05.523248 2777 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Sep 28 21:58:05 dist-test-slave-k6lz krb5kdc[21881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632685, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:58:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:58:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab.
Entry for principal kudu/127.2.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab.
I20260928 21:58:05.642340 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:43805
--webserver_interface=127.2.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab
--principal=kudu/127.2.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:45861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.126:43805
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:58:05.721940 21894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:58:05.722162 21894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:58:05.722183 21894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:58:05.723407 21894 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:58:05.723448 21894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:58:05.723459 21894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:58:05.723471 21894 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:58:05.723485 21894 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:58:05.725004 21894 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:45861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.126:43805
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab
--principal=kudu/127.2.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:43805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:58:05.725224 21894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:58:05.725469 21894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:58:05.730288 21902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:05.730300 21900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:05.730350 21899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:05.730571 21894 server_base.cc:1034] running on GCE node
I20260928 21:58:05.730787 21894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:58:05.731050 21894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:58:05.732187 21894 hybrid_clock.cc:648] HybridClock initialized: now 1790632685732176 us; error 26 us; skew 500 ppm
Sep 28 21:58:05 dist-test-slave-k6lz krb5kdc[21881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632685, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:58:05.734529 21894 init.cc:377] Logged in from keytab as kudu/127.2.182.126@KRBTEST.COM (short username kudu)
I20260928 21:58:05.735011 21894 webserver.cc:460] Webserver started at http://127.2.182.126:40137/ using document root <none> and password file <none>
I20260928 21:58:05.735217 21894 fs_manager.cc:359] Metadata directory not provided
I20260928 21:58:05.735265 21894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:58:05.735412 21894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:58:05.736480 21894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/minicluster-data/master-0/data/instance:
uuid: "37a27d7dd1cd49038bb296c2bb86f989"
format_stamp: "Formatted at 2026-09-28 21:58:05 on dist-test-slave-k6lz"
server_key: "58625669880c1004c9a3fc5cba6d25ad"
server_key_iv: "3493c37a5a3b3d5e628dcb0eb5d3ea9c"
server_key_version: "encryptionkey@0"
I20260928 21:58:05.736809 21894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/minicluster-data/master-0/wal/instance:
uuid: "37a27d7dd1cd49038bb296c2bb86f989"
format_stamp: "Formatted at 2026-09-28 21:58:05 on dist-test-slave-k6lz"
server_key: "58625669880c1004c9a3fc5cba6d25ad"
server_key_iv: "3493c37a5a3b3d5e628dcb0eb5d3ea9c"
server_key_version: "encryptionkey@0"
I20260928 21:58:05.738142 21894 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:58:05.738917 21909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:05.739097 21894 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:58:05.739180 21894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/minicluster-data/master-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/minicluster-data/master-0/wal
uuid: "37a27d7dd1cd49038bb296c2bb86f989"
format_stamp: "Formatted at 2026-09-28 21:58:05 on dist-test-slave-k6lz"
server_key: "58625669880c1004c9a3fc5cba6d25ad"
server_key_iv: "3493c37a5a3b3d5e628dcb0eb5d3ea9c"
server_key_version: "encryptionkey@0"
I20260928 21:58:05.739243 21894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:58:05.760282 21894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:58:05.762354 21894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:58:05.762490 21894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:58:05.766484 21894 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.126:43805
I20260928 21:58:05.766525 21961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.126:43805 every 8 connection(s)
I20260928 21:58:05.766851 21894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
I20260928 21:58:05.767042 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 21894
I20260928 21:58:05.767150 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/minicluster-data/master-0/wal/instance
I20260928 21:58:05.767349 2777 external_mini_cluster.cc:1428] Setting key 72487c43a2263a2ee389d67690470f87
I20260928 21:58:05.767823 21962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:58:05.770833 21962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37a27d7dd1cd49038bb296c2bb86f989: Bootstrap starting.
I20260928 21:58:05.771435 21962 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37a27d7dd1cd49038bb296c2bb86f989: Neither blocks nor log segments found. Creating new log.
I20260928 21:58:05.771735 21962 log.cc:824] T 00000000000000000000000000000000 P 37a27d7dd1cd49038bb296c2bb86f989: Log is configured to *not* fsync() on all Append() calls
I20260928 21:58:05.772471 21962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37a27d7dd1cd49038bb296c2bb86f989: No bootstrap required, opened a new log
Sep 28 21:58:05 dist-test-slave-k6lz krb5kdc[21881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632685, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:58:05.774322 21962 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37a27d7dd1cd49038bb296c2bb86f989 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37a27d7dd1cd49038bb296c2bb86f989" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43805 } }
I20260928 21:58:05.774505 21962 raft_consensus.cc:374] T 00000000000000000000000000000000 P 37a27d7dd1cd49038bb296c2bb86f989 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:58:05.774561 21962 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37a27d7dd1cd49038bb296c2bb86f989 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37a27d7dd1cd49038bb296c2bb86f989, State: Initialized, Role: FOLLOWER
I20260928 21:58:05.774688 21962 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37a27d7dd1cd49038bb296c2bb86f989 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37a27d7dd1cd49038bb296c2bb86f989" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43805 } }
I20260928 21:58:05.774767 21962 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37a27d7dd1cd49038bb296c2bb86f989 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:58:05.774798 21962 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37a27d7dd1cd49038bb296c2bb86f989 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:58:05.774848 21962 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37a27d7dd1cd49038bb296c2bb86f989 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:58:05.775556 21962 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37a27d7dd1cd49038bb296c2bb86f989 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37a27d7dd1cd49038bb296c2bb86f989" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43805 } }
I20260928 21:58:05.775660 21962 leader_election.cc:302] T 00000000000000000000000000000000 P 37a27d7dd1cd49038bb296c2bb86f989 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37a27d7dd1cd49038bb296c2bb86f989; no voters:
I20260928 21:58:05.775806 21962 leader_election.cc:288] T 00000000000000000000000000000000 P 37a27d7dd1cd49038bb296c2bb86f989 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:58:05.775859 21967 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37a27d7dd1cd49038bb296c2bb86f989 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:58:05.776026 21967 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37a27d7dd1cd49038bb296c2bb86f989 [term 1 LEADER]: Becoming Leader. State: Replica: 37a27d7dd1cd49038bb296c2bb86f989, State: Running, Role: LEADER
I20260928 21:58:05.776059 21962 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37a27d7dd1cd49038bb296c2bb86f989 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:58:05.776114 21967 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37a27d7dd1cd49038bb296c2bb86f989 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37a27d7dd1cd49038bb296c2bb86f989" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43805 } }
I20260928 21:58:05.776456 21969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37a27d7dd1cd49038bb296c2bb86f989 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37a27d7dd1cd49038bb296c2bb86f989. Latest consensus state: current_term: 1 leader_uuid: "37a27d7dd1cd49038bb296c2bb86f989" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37a27d7dd1cd49038bb296c2bb86f989" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43805 } } }
I20260928 21:58:05.776539 21969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37a27d7dd1cd49038bb296c2bb86f989 [sys.catalog]: This master's current role is: LEADER
I20260928 21:58:05.776443 21968 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37a27d7dd1cd49038bb296c2bb86f989 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "37a27d7dd1cd49038bb296c2bb86f989" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37a27d7dd1cd49038bb296c2bb86f989" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 43805 } } }
I20260928 21:58:05.776680 21968 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37a27d7dd1cd49038bb296c2bb86f989 [sys.catalog]: This master's current role is: LEADER
I20260928 21:58:05.776991 21972 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:58:05.777526 21972 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:58:05.778134 21965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:05.767837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49460 (local address 127.2.182.126:43805)
0928 21:58:05.768015 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:05.768024 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:05.768062 (+ 38us) server_negotiation.cc:407] Connection header received
0928 21:58:05.768548 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:05.768556 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:05.768617 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:05.768761 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:05.769315 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:05.770664 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:05.771408 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:05.771984 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:05.772069 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:05.774166 (+ 2097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:05.774170 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:05.774170 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:05.774183 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:05.776087 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:05.776512 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:05.776514 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:05.776515 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:05.776619 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:05.776795 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:05.776796 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:05.776796 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:05.777115 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:05.777716 (+ 601us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:05.777792 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 21:58:05.777931 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":99,"threads_started":1}
I20260928 21:58:05.779112 21972 catalog_manager.cc:1347] Generated new cluster ID: 3f90e856231e4a018ead797d9577fd10
I20260928 21:58:05.779163 21972 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:58:05.783505 21972 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:58:05.784015 21972 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:58:05.794624 21972 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 37a27d7dd1cd49038bb296c2bb86f989: Generated new TSK 0
I20260928 21:58:05.794839 21972 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.182.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab.
Entry for principal kudu/127.2.182.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab.
I20260928 21:58:05.824616 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.65:0
--local_ip_for_outbound_sockets=127.2.182.65
--webserver_interface=127.2.182.65
--webserver_port=0
--tserver_master_addrs=127.2.182.126:43805
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab
--principal=kudu/127.2.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:45861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:58:05.926569 21988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:58:05.926779 21988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:58:05.926811 21988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:58:05.928049 21988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:58:05.928095 21988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:58:05.928131 21988 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.65
I20260928 21:58:05.929589 21988 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:45861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.65.keytab
--principal=kudu/127.2.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.182.65
--webserver_port=0
--tserver_master_addrs=127.2.182.126:43805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21988
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.65
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:58:05.929821 21988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:58:05.930027 21988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:58:05.935132 21993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:05.935110 21996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:05.935112 21994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:05.935318 21988 server_base.cc:1034] running on GCE node
I20260928 21:58:05.935591 21988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:58:05.935830 21988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:58:05.936995 21988 hybrid_clock.cc:648] HybridClock initialized: now 1790632685936985 us; error 31 us; skew 500 ppm
Sep 28 21:58:05 dist-test-slave-k6lz krb5kdc[21881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632685, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:58:05.939491 21988 init.cc:377] Logged in from keytab as kudu/127.2.182.65@KRBTEST.COM (short username kudu)
I20260928 21:58:05.940006 21988 webserver.cc:460] Webserver started at http://127.2.182.65:41937/ using document root <none> and password file <none>
I20260928 21:58:05.940199 21988 fs_manager.cc:359] Metadata directory not provided
I20260928 21:58:05.940245 21988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:58:05.940369 21988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:58:05.941381 21988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/minicluster-data/ts-0/data/instance:
uuid: "eaed6ab8f1a94a5b921e25fab99bd4f8"
format_stamp: "Formatted at 2026-09-28 21:58:05 on dist-test-slave-k6lz"
server_key: "08aa12cfb6e3387df7cc90f6c1b02369"
server_key_iv: "235f4654fa5cb66424ca65d926491b2f"
server_key_version: "encryptionkey@0"
I20260928 21:58:05.941718 21988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/minicluster-data/ts-0/wal/instance:
uuid: "eaed6ab8f1a94a5b921e25fab99bd4f8"
format_stamp: "Formatted at 2026-09-28 21:58:05 on dist-test-slave-k6lz"
server_key: "08aa12cfb6e3387df7cc90f6c1b02369"
server_key_iv: "235f4654fa5cb66424ca65d926491b2f"
server_key_version: "encryptionkey@0"
I20260928 21:58:05.942955 21988 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 21:58:05.943704 22003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:05.943849 21988 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:05.943936 21988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/minicluster-data/ts-0/wal
uuid: "eaed6ab8f1a94a5b921e25fab99bd4f8"
format_stamp: "Formatted at 2026-09-28 21:58:05 on dist-test-slave-k6lz"
server_key: "08aa12cfb6e3387df7cc90f6c1b02369"
server_key_iv: "235f4654fa5cb66424ca65d926491b2f"
server_key_version: "encryptionkey@0"
I20260928 21:58:05.943996 21988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:58:05.962543 21988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:58:05.964609 21988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:58:05.964756 21988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:58:05.964974 21988 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:58:05.965268 21988 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:58:05.965301 21988 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:05.965330 21988 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:58:05.965353 21988 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:05.971453 21988 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.65:37023
I20260928 21:58:05.971511 22116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.65:37023 every 8 connection(s)
I20260928 21:58:05.971818 21988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
Sep 28 21:58:05 dist-test-slave-k6lz krb5kdc[21881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632685, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.65@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:58:05.980530 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 21988
I20260928 21:58:05.980628 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/minicluster-data/ts-0/wal/instance
I20260928 21:58:05.980820 2777 external_mini_cluster.cc:1428] Setting key 228038e59cc91257dde6badceb9a0943
I20260928 21:58:05.980983 21965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:05.972641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.65:50311 (local address 127.2.182.126:43805)
0928 21:58:05.972694 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:05.972695 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:05.972706 (+ 11us) server_negotiation.cc:407] Connection header received
0928 21:58:05.973205 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:05.973206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:05.973218 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:05.973279 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:05.974208 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:05.974765 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:05.975903 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:05.976143 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:05.976242 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:05.978354 (+ 2112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:05.978355 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:05.978356 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:05.978363 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:05.979579 (+ 1216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:05.979979 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:05.979981 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:05.979981 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:05.980035 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:05.980227 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:05.980228 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:05.980228 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:05.980339 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:05.980403 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:05.980802 (+ 399us) server_negotiation.cc:299] Negotiation successful
0928 21:58:05.980896 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 21:58:05.981164 22119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:05.972441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:43805 (local address 127.2.182.65:50311)
0928 21:58:05.972596 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:05.972639 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:05.973094 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:05.973330 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:05.973333 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:05.973887 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:58:05.974086 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:05.974092 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:05.974874 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:05.974875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:05.975817 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:05.975820 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:05.976253 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:05.976254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:05.976360 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:05.976855 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:58:05.976863 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:58:05.978259 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:58:05.979674 (+ 1415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:05.979681 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:05.979683 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:05.979905 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:05.980088 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:05.980089 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:05.980089 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:05.980182 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:05.980420 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:58:05.980428 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:58:05.980652 (+ 224us) client_negotiation.cc:769] Sending connection context
0928 21:58:05.980763 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 21:58:05.980928 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":65,"threads_started":1}
I20260928 21:58:05.981645 22117 heartbeater.cc:347] Connected to a master server at 127.2.182.126:43805
I20260928 21:58:05.981740 22117 heartbeater.cc:464] Registering TS with master...
I20260928 21:58:05.981911 22117 heartbeater.cc:511] Master 127.2.182.126:43805 requested a full tablet report, sending...
I20260928 21:58:05.982430 21925 ts_manager.cc:194] Registered new tserver with Master: eaed6ab8f1a94a5b921e25fab99bd4f8 (127.2.182.65:37023)
I20260928 21:58:05.983210 21925 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.65@KRBTEST.COM'} at 127.2.182.65:50311
WARNING: no policy specified for kudu/127.2.182.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab.
Entry for principal kudu/127.2.182.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab.
I20260928 21:58:06.007623 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.66:0
--local_ip_for_outbound_sockets=127.2.182.66
--webserver_interface=127.2.182.66
--webserver_port=0
--tserver_master_addrs=127.2.182.126:43805
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab
--principal=kudu/127.2.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:45861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:58:06.088765 22122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:58:06.088992 22122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:58:06.089016 22122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:58:06.090193 22122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:58:06.090245 22122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:58:06.090271 22122 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.66
I20260928 21:58:06.091794 22122 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:45861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.66.keytab
--principal=kudu/127.2.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.182.66
--webserver_port=0
--tserver_master_addrs=127.2.182.126:43805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22122
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.66
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:58:06.092028 22122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:58:06.092260 22122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:58:06.097276 22128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:06.097294 22122 server_base.cc:1034] running on GCE node
W20260928 21:58:06.097276 22130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:06.097306 22127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:06.097746 22122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:58:06.097975 22122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:58:06.099114 22122 hybrid_clock.cc:648] HybridClock initialized: now 1790632686099094 us; error 32 us; skew 500 ppm
Sep 28 21:58:06 dist-test-slave-k6lz krb5kdc[21881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632686, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:58:06.101719 22122 init.cc:377] Logged in from keytab as kudu/127.2.182.66@KRBTEST.COM (short username kudu)
I20260928 21:58:06.102164 22122 webserver.cc:460] Webserver started at http://127.2.182.66:34225/ using document root <none> and password file <none>
I20260928 21:58:06.102344 22122 fs_manager.cc:359] Metadata directory not provided
I20260928 21:58:06.102384 22122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:58:06.102468 22122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:58:06.103574 22122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/minicluster-data/ts-1/data/instance:
uuid: "b856e73f795741b8a27cf02b90f8215f"
format_stamp: "Formatted at 2026-09-28 21:58:06 on dist-test-slave-k6lz"
server_key: "8a8197fc7d531d51081bcd38c938f72f"
server_key_iv: "1c5f9c100d86d760185820e1d9049939"
server_key_version: "encryptionkey@0"
I20260928 21:58:06.104055 22122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/minicluster-data/ts-1/wal/instance:
uuid: "b856e73f795741b8a27cf02b90f8215f"
format_stamp: "Formatted at 2026-09-28 21:58:06 on dist-test-slave-k6lz"
server_key: "8a8197fc7d531d51081bcd38c938f72f"
server_key_iv: "1c5f9c100d86d760185820e1d9049939"
server_key_version: "encryptionkey@0"
I20260928 21:58:06.105327 22122 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:58:06.106101 22137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:06.106307 22122 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:58:06.106389 22122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/minicluster-data/ts-1/wal
uuid: "b856e73f795741b8a27cf02b90f8215f"
format_stamp: "Formatted at 2026-09-28 21:58:06 on dist-test-slave-k6lz"
server_key: "8a8197fc7d531d51081bcd38c938f72f"
server_key_iv: "1c5f9c100d86d760185820e1d9049939"
server_key_version: "encryptionkey@0"
I20260928 21:58:06.106446 22122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:58:06.123060 22122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:58:06.125401 22122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:58:06.125519 22122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:58:06.125804 22122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:58:06.126073 22122 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:58:06.126101 22122 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:06.126127 22122 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:58:06.126144 22122 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:06.131603 22122 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.66:44371
I20260928 21:58:06.131676 22250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.66:44371 every 8 connection(s)
I20260928 21:58:06.131946 22122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
I20260928 21:58:06.132457 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 22122
I20260928 21:58:06.132570 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/minicluster-data/ts-1/wal/instance
I20260928 21:58:06.132779 2777 external_mini_cluster.cc:1428] Setting key a0abbdd65779377b2231e712e312dd05
Sep 28 21:58:06 dist-test-slave-k6lz krb5kdc[21881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632686, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.66@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.182.67@KRBTEST.COM; defaulting to no policy
I20260928 21:58:06.141508 21965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:06.132809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.66:50377 (local address 127.2.182.126:43805)
0928 21:58:06.132872 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:06.132874 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:06.133093 (+ 219us) server_negotiation.cc:407] Connection header received
0928 21:58:06.133647 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:06.133648 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:06.133660 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:06.133707 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:06.134749 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:06.135246 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:06.136275 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:06.136477 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:06.136533 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:06.138602 (+ 2069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:06.138604 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:06.138604 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:06.138612 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:06.140103 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:06.140591 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:06.140592 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:06.140593 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:06.140647 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:06.140913 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:06.140914 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:06.140914 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:06.141036 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:06.141098 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:06.141337 (+ 239us) server_negotiation.cc:299] Negotiation successful
0928 21:58:06.141410 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 21:58:06.141753 22253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:06.132778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:43805 (local address 127.2.182.66:50377)
0928 21:58:06.132966 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:06.133022 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:06.133558 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:06.133776 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:06.133780 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:06.134366 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:58:06.134651 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:06.134657 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:06.135352 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:06.135353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:06.136191 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:06.136193 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:06.136548 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:06.136548 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:06.136676 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:06.137164 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:58:06.137173 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:58:06.138508 (+ 1335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:58:06.140179 (+ 1671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:06.140187 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:06.140190 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:06.140510 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:06.140703 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:06.140704 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:06.140704 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:06.140813 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:06.141099 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:58:06.141105 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:58:06.141231 (+ 126us) client_negotiation.cc:769] Sending connection context
0928 21:58:06.141317 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 21:58:06.141524 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":91,"threads_started":1}
I20260928 21:58:06.142241 22251 heartbeater.cc:347] Connected to a master server at 127.2.182.126:43805
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.67@KRBTEST.COM" created.
I20260928 21:58:06.142338 22251 heartbeater.cc:464] Registering TS with master...
I20260928 21:58:06.142515 22251 heartbeater.cc:511] Master 127.2.182.126:43805 requested a full tablet report, sending...
I20260928 21:58:06.142877 21925 ts_manager.cc:194] Registered new tserver with Master: b856e73f795741b8a27cf02b90f8215f (127.2.182.66:44371)
I20260928 21:58:06.143378 21925 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.66@KRBTEST.COM'} at 127.2.182.66:50377
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab.
Entry for principal kudu/127.2.182.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab.
I20260928 21:58:06.158143 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.67:0
--local_ip_for_outbound_sockets=127.2.182.67
--webserver_interface=127.2.182.67
--webserver_port=0
--tserver_master_addrs=127.2.182.126:43805
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab
--principal=kudu/127.2.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:45861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:58:06.237833 22256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:58:06.238255 22256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:58:06.238294 22256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:58:06.239908 22256 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:58:06.239972 22256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:58:06.240012 22256 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.67
I20260928 21:58:06.241585 22256 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:45861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.67.keytab
--principal=kudu/127.2.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.182.67
--webserver_port=0
--tserver_master_addrs=127.2.182.126:43805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22256
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.67
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:58:06.241860 22256 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:58:06.242089 22256 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:58:06.247155 22262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:06.247275 22256 server_base.cc:1034] running on GCE node
W20260928 21:58:06.247316 22261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:06.247359 22264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:06.247597 22256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:58:06.247834 22256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:58:06.248997 22256 hybrid_clock.cc:648] HybridClock initialized: now 1790632686248981 us; error 31 us; skew 500 ppm
Sep 28 21:58:06 dist-test-slave-k6lz krb5kdc[21881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632686, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:58:06.251793 22256 init.cc:377] Logged in from keytab as kudu/127.2.182.67@KRBTEST.COM (short username kudu)
I20260928 21:58:06.252223 22256 webserver.cc:460] Webserver started at http://127.2.182.67:43953/ using document root <none> and password file <none>
I20260928 21:58:06.252449 22256 fs_manager.cc:359] Metadata directory not provided
I20260928 21:58:06.252490 22256 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:58:06.252600 22256 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:58:06.253612 22256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/minicluster-data/ts-2/data/instance:
uuid: "016cc66224c64dd79fc10904fd06b3e9"
format_stamp: "Formatted at 2026-09-28 21:58:06 on dist-test-slave-k6lz"
server_key: "af2454831f6ac7883a721d2824e1548e"
server_key_iv: "280fe6d93445ca76ab9391d36942adc2"
server_key_version: "encryptionkey@0"
I20260928 21:58:06.253926 22256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/minicluster-data/ts-2/wal/instance:
uuid: "016cc66224c64dd79fc10904fd06b3e9"
format_stamp: "Formatted at 2026-09-28 21:58:06 on dist-test-slave-k6lz"
server_key: "af2454831f6ac7883a721d2824e1548e"
server_key_iv: "280fe6d93445ca76ab9391d36942adc2"
server_key_version: "encryptionkey@0"
I20260928 21:58:06.255100 22256 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 21:58:06.255928 22271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:06.256104 22256 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:06.256174 22256 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/minicluster-data/ts-2/wal
uuid: "016cc66224c64dd79fc10904fd06b3e9"
format_stamp: "Formatted at 2026-09-28 21:58:06 on dist-test-slave-k6lz"
server_key: "af2454831f6ac7883a721d2824e1548e"
server_key_iv: "280fe6d93445ca76ab9391d36942adc2"
server_key_version: "encryptionkey@0"
I20260928 21:58:06.256237 22256 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:58:06.264380 22256 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:58:06.266077 22256 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:58:06.266199 22256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:58:06.266388 22256 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:58:06.266664 22256 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:58:06.266697 22256 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:06.266726 22256 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:58:06.266749 22256 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:06.272087 22256 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.67:42493
I20260928 21:58:06.272136 22384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.67:42493 every 8 connection(s)
I20260928 21:58:06.272464 22256 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
I20260928 21:58:06.272927 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 22256
I20260928 21:58:06.273069 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/minicluster-data/ts-2/wal/instance
I20260928 21:58:06.273275 2777 external_mini_cluster.cc:1428] Setting key 850e7ea93540eda2105837020ecb7ea4
Sep 28 21:58:06 dist-test-slave-k6lz krb5kdc[21881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632686, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.67@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:58:06.281966 21965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:06.273098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.67:42463 (local address 127.2.182.126:43805)
0928 21:58:06.273148 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:06.273150 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:06.273361 (+ 211us) server_negotiation.cc:407] Connection header received
0928 21:58:06.274057 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:06.274058 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:06.274071 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:06.274124 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:06.275050 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:06.275621 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:06.276693 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:06.276979 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:06.277037 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:06.279282 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:06.279283 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:06.279283 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:06.279290 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:06.280574 (+ 1284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:06.280996 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:06.280997 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:06.280998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:06.281050 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:06.281314 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:06.281315 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:06.281315 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:06.281494 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:06.281548 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:06.281832 (+ 284us) server_negotiation.cc:299] Negotiation successful
0928 21:58:06.281901 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 21:58:06.282132 22387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:06.273076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:43805 (local address 127.2.182.67:42463)
0928 21:58:06.273232 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:06.273279 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:06.273967 (+ 688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:06.274194 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:06.274198 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:06.274791 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:58:06.274980 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:06.274985 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:06.275718 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:06.275719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:06.276616 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:06.276619 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:06.277068 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:06.277069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:06.277174 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:06.277628 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:58:06.277638 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:58:06.279178 (+ 1540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:58:06.280652 (+ 1474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:06.280659 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:06.280660 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:06.280926 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:06.281140 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:06.281141 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:06.281142 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:06.281249 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:06.281589 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:58:06.281595 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:58:06.281711 (+ 116us) client_negotiation.cc:769] Sending connection context
0928 21:58:06.281800 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 21:58:06.281922 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":81,"threads_started":1}
I20260928 21:58:06.282647 22385 heartbeater.cc:347] Connected to a master server at 127.2.182.126:43805
I20260928 21:58:06.282745 22385 heartbeater.cc:464] Registering TS with master...
I20260928 21:58:06.282933 22385 heartbeater.cc:511] Master 127.2.182.126:43805 requested a full tablet report, sending...
I20260928 21:58:06.283397 21925 ts_manager.cc:194] Registered new tserver with Master: 016cc66224c64dd79fc10904fd06b3e9 (127.2.182.67:42493)
I20260928 21:58:06.283751 21925 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.182.67@KRBTEST.COM'} at 127.2.182.67:42463
I20260928 21:58:06.284996 2777 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:58:06.291440 21965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:06.286184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49468 (local address 127.2.182.126:43805)
0928 21:58:06.286222 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:06.286223 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:06.286288 (+ 65us) server_negotiation.cc:407] Connection header received
0928 21:58:06.286335 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:06.286336 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:06.286344 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:06.286389 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:06.286858 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:06.287370 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:06.288066 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:06.288247 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:06.288298 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:06.289327 (+ 1029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:06.289329 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:06.289329 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:06.289335 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:06.290560 (+ 1225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:06.290843 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:06.290844 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:06.290844 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:06.290892 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:06.291023 (+ 131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:06.291023 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:06.291023 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:06.291152 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:06.291223 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:06.291290 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 21:58:06.291375 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 21:58:06.292889 21925 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49468:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:58:06.294450 21925 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:58:06.300812 22396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:06.296584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.66:44371 (local address 127.0.0.1:55340)
0928 21:58:06.296719 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:06.296756 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:06.296815 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:06.297213 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:06.297214 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:06.297216 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:58:06.297291 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:06.297293 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:06.298716 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:06.298717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:06.299625 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:06.299628 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:06.300525 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:06.300526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:06.300592 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:06.300603 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 21:58:06.300673 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 21:58:06.300727 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":4,"thread_start_us":40,"threads_started":1}
I20260928 21:58:06.300824 22401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:06.296898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55340 (local address 127.2.182.66:44371)
0928 21:58:06.297061 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:06.297064 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:06.297074 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:58:06.297110 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:06.297112 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:06.297126 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:06.297197 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:58:06.297572 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:06.298630 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:06.299693 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:06.300438 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:06.300514 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:06.300573 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:06.300702 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 21:58:06.300745 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":67,"threads_started":1}
I20260928 21:58:06.300984 22400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:06.296868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57966 (local address 127.2.182.65:37023)
0928 21:58:06.297057 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:06.297062 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:06.297358 (+ 296us) server_negotiation.cc:407] Connection header received
0928 21:58:06.297403 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:06.297406 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:06.297420 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:06.297491 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:58:06.297633 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:06.298379 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:06.299863 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:06.300594 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:06.300654 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:06.300703 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:06.300867 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 21:58:06.300912 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":77,"threads_started":1}
I20260928 21:58:06.301137 22399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:06.296754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.65:37023 (local address 127.0.0.1:57966)
0928 21:58:06.297290 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:06.297300 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:06.297348 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:06.297501 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:06.297501 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:06.297503 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:58:06.297559 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:06.297560 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:06.298463 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:06.298465 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:06.299770 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:06.299773 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:06.300670 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:06.300671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:06.300733 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:06.300737 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:58:06.300782 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 21:58:06.300815 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"mutex_wait_us":2,"thread_start_us":65,"threads_started":1}
I20260928 21:58:06.301875 22397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:06.296617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.67:42493 (local address 127.0.0.1:49268)
0928 21:58:06.296886 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:06.296897 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:06.296954 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:06.297209 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:06.297211 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:06.297214 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:58:06.297310 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:06.297312 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:06.298649 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:06.298650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:06.299772 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:06.299773 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:06.301641 (+ 1868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:06.301642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:06.301710 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:06.301715 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 21:58:06.301770 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 21:58:06.301810 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":11,"thread_start_us":48,"threads_started":1}
I20260928 21:58:06.301895 22185 tablet_service.cc:1467] Processing CreateTablet for tablet 0b34f03603394d99b0d8b9947b820ac5 (DEFAULT_TABLE table=test-table [id=c758c1a65d104f779fbc1705a7e0524d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:58:06.302062 22398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:06.296712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49268 (local address 127.2.182.67:42493)
0928 21:58:06.297069 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:06.297072 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:06.297079 (+ 7us) server_negotiation.cc:407] Connection header received
0928 21:58:06.297111 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:06.297113 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:06.297125 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:06.297190 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:58:06.297398 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:06.298544 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:06.300738 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:06.301554 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:06.301626 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:06.301675 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:06.301852 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 21:58:06.301952 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":94,"threads_started":1}
I20260928 21:58:06.302248 22185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b34f03603394d99b0d8b9947b820ac5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:58:06.302637 22319 tablet_service.cc:1467] Processing CreateTablet for tablet 0b34f03603394d99b0d8b9947b820ac5 (DEFAULT_TABLE table=test-table [id=c758c1a65d104f779fbc1705a7e0524d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:58:06.302876 22319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b34f03603394d99b0d8b9947b820ac5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:58:06.303783 22051 tablet_service.cc:1467] Processing CreateTablet for tablet 0b34f03603394d99b0d8b9947b820ac5 (DEFAULT_TABLE table=test-table [id=c758c1a65d104f779fbc1705a7e0524d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:58:06.304103 22051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b34f03603394d99b0d8b9947b820ac5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:58:06.305873 22403 tablet_bootstrap.cc:492] T 0b34f03603394d99b0d8b9947b820ac5 P 016cc66224c64dd79fc10904fd06b3e9: Bootstrap starting.
I20260928 21:58:06.305946 22402 tablet_bootstrap.cc:492] T 0b34f03603394d99b0d8b9947b820ac5 P b856e73f795741b8a27cf02b90f8215f: Bootstrap starting.
I20260928 21:58:06.306574 22403 tablet_bootstrap.cc:654] T 0b34f03603394d99b0d8b9947b820ac5 P 016cc66224c64dd79fc10904fd06b3e9: Neither blocks nor log segments found. Creating new log.
I20260928 21:58:06.306572 22402 tablet_bootstrap.cc:654] T 0b34f03603394d99b0d8b9947b820ac5 P b856e73f795741b8a27cf02b90f8215f: Neither blocks nor log segments found. Creating new log.
I20260928 21:58:06.307055 22403 log.cc:824] T 0b34f03603394d99b0d8b9947b820ac5 P 016cc66224c64dd79fc10904fd06b3e9: Log is configured to *not* fsync() on all Append() calls
I20260928 21:58:06.307091 22402 log.cc:824] T 0b34f03603394d99b0d8b9947b820ac5 P b856e73f795741b8a27cf02b90f8215f: Log is configured to *not* fsync() on all Append() calls
I20260928 21:58:06.307363 22404 tablet_bootstrap.cc:492] T 0b34f03603394d99b0d8b9947b820ac5 P eaed6ab8f1a94a5b921e25fab99bd4f8: Bootstrap starting.
I20260928 21:58:06.307811 22403 tablet_bootstrap.cc:492] T 0b34f03603394d99b0d8b9947b820ac5 P 016cc66224c64dd79fc10904fd06b3e9: No bootstrap required, opened a new log
I20260928 21:58:06.307885 22403 ts_tablet_manager.cc:1397] T 0b34f03603394d99b0d8b9947b820ac5 P 016cc66224c64dd79fc10904fd06b3e9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:58:06.307933 22402 tablet_bootstrap.cc:492] T 0b34f03603394d99b0d8b9947b820ac5 P b856e73f795741b8a27cf02b90f8215f: No bootstrap required, opened a new log
I20260928 21:58:06.307994 22402 ts_tablet_manager.cc:1397] T 0b34f03603394d99b0d8b9947b820ac5 P b856e73f795741b8a27cf02b90f8215f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:58:06.308089 22404 tablet_bootstrap.cc:654] T 0b34f03603394d99b0d8b9947b820ac5 P eaed6ab8f1a94a5b921e25fab99bd4f8: Neither blocks nor log segments found. Creating new log.
I20260928 21:58:06.308590 22404 log.cc:824] T 0b34f03603394d99b0d8b9947b820ac5 P eaed6ab8f1a94a5b921e25fab99bd4f8: Log is configured to *not* fsync() on all Append() calls
I20260928 21:58:06.309363 22404 tablet_bootstrap.cc:492] T 0b34f03603394d99b0d8b9947b820ac5 P eaed6ab8f1a94a5b921e25fab99bd4f8: No bootstrap required, opened a new log
I20260928 21:58:06.309430 22404 ts_tablet_manager.cc:1397] T 0b34f03603394d99b0d8b9947b820ac5 P eaed6ab8f1a94a5b921e25fab99bd4f8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 21:58:06.309749 22402 raft_consensus.cc:348] T 0b34f03603394d99b0d8b9947b820ac5 P b856e73f795741b8a27cf02b90f8215f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "016cc66224c64dd79fc10904fd06b3e9" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 42493 } } peers { permanent_uuid: "b856e73f795741b8a27cf02b90f8215f" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 44371 } } peers { permanent_uuid: "eaed6ab8f1a94a5b921e25fab99bd4f8" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 37023 } }
I20260928 21:58:06.309980 22402 raft_consensus.cc:374] T 0b34f03603394d99b0d8b9947b820ac5 P b856e73f795741b8a27cf02b90f8215f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:58:06.309877 22403 raft_consensus.cc:348] T 0b34f03603394d99b0d8b9947b820ac5 P 016cc66224c64dd79fc10904fd06b3e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "016cc66224c64dd79fc10904fd06b3e9" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 42493 } } peers { permanent_uuid: "b856e73f795741b8a27cf02b90f8215f" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 44371 } } peers { permanent_uuid: "eaed6ab8f1a94a5b921e25fab99bd4f8" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 37023 } }
I20260928 21:58:06.310037 22402 raft_consensus.cc:729] T 0b34f03603394d99b0d8b9947b820ac5 P b856e73f795741b8a27cf02b90f8215f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b856e73f795741b8a27cf02b90f8215f, State: Initialized, Role: FOLLOWER
I20260928 21:58:06.310047 22403 raft_consensus.cc:374] T 0b34f03603394d99b0d8b9947b820ac5 P 016cc66224c64dd79fc10904fd06b3e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:58:06.310078 22403 raft_consensus.cc:729] T 0b34f03603394d99b0d8b9947b820ac5 P 016cc66224c64dd79fc10904fd06b3e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 016cc66224c64dd79fc10904fd06b3e9, State: Initialized, Role: FOLLOWER
I20260928 21:58:06.310154 22402 consensus_queue.cc:260] T 0b34f03603394d99b0d8b9947b820ac5 P b856e73f795741b8a27cf02b90f8215f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "016cc66224c64dd79fc10904fd06b3e9" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 42493 } } peers { permanent_uuid: "b856e73f795741b8a27cf02b90f8215f" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 44371 } } peers { permanent_uuid: "eaed6ab8f1a94a5b921e25fab99bd4f8" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 37023 } }
I20260928 21:58:06.310182 22403 consensus_queue.cc:260] T 0b34f03603394d99b0d8b9947b820ac5 P 016cc66224c64dd79fc10904fd06b3e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "016cc66224c64dd79fc10904fd06b3e9" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 42493 } } peers { permanent_uuid: "b856e73f795741b8a27cf02b90f8215f" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 44371 } } peers { permanent_uuid: "eaed6ab8f1a94a5b921e25fab99bd4f8" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 37023 } }
I20260928 21:58:06.310444 22403 ts_tablet_manager.cc:1428] T 0b34f03603394d99b0d8b9947b820ac5 P 016cc66224c64dd79fc10904fd06b3e9: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 21:58:06.310444 22402 ts_tablet_manager.cc:1428] T 0b34f03603394d99b0d8b9947b820ac5 P b856e73f795741b8a27cf02b90f8215f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 21:58:06.310463 22385 heartbeater.cc:503] Master 127.2.182.126:43805 was elected leader, sending a full tablet report...
I20260928 21:58:06.310566 22251 heartbeater.cc:503] Master 127.2.182.126:43805 was elected leader, sending a full tablet report...
I20260928 21:58:06.311560 22404 raft_consensus.cc:348] T 0b34f03603394d99b0d8b9947b820ac5 P eaed6ab8f1a94a5b921e25fab99bd4f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "016cc66224c64dd79fc10904fd06b3e9" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 42493 } } peers { permanent_uuid: "b856e73f795741b8a27cf02b90f8215f" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 44371 } } peers { permanent_uuid: "eaed6ab8f1a94a5b921e25fab99bd4f8" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 37023 } }
I20260928 21:58:06.311721 22404 raft_consensus.cc:374] T 0b34f03603394d99b0d8b9947b820ac5 P eaed6ab8f1a94a5b921e25fab99bd4f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:58:06.311765 22404 raft_consensus.cc:729] T 0b34f03603394d99b0d8b9947b820ac5 P eaed6ab8f1a94a5b921e25fab99bd4f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaed6ab8f1a94a5b921e25fab99bd4f8, State: Initialized, Role: FOLLOWER
I20260928 21:58:06.311879 22404 consensus_queue.cc:260] T 0b34f03603394d99b0d8b9947b820ac5 P eaed6ab8f1a94a5b921e25fab99bd4f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "016cc66224c64dd79fc10904fd06b3e9" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 42493 } } peers { permanent_uuid: "b856e73f795741b8a27cf02b90f8215f" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 44371 } } peers { permanent_uuid: "eaed6ab8f1a94a5b921e25fab99bd4f8" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 37023 } }
I20260928 21:58:06.312124 22404 ts_tablet_manager.cc:1428] T 0b34f03603394d99b0d8b9947b820ac5 P eaed6ab8f1a94a5b921e25fab99bd4f8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 21:58:06.312178 22117 heartbeater.cc:503] Master 127.2.182.126:43805 was elected leader, sending a full tablet report...
W20260928 21:58:06.382634 22252 tablet.cc:2369] T 0b34f03603394d99b0d8b9947b820ac5 P b856e73f795741b8a27cf02b90f8215f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 21:58:06.472537 22118 tablet.cc:2369] T 0b34f03603394d99b0d8b9947b820ac5 P eaed6ab8f1a94a5b921e25fab99bd4f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:58:06.498869 22410 raft_consensus.cc:482] T 0b34f03603394d99b0d8b9947b820ac5 P eaed6ab8f1a94a5b921e25fab99bd4f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:58:06.499012 22410 raft_consensus.cc:504] T 0b34f03603394d99b0d8b9947b820ac5 P eaed6ab8f1a94a5b921e25fab99bd4f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "016cc66224c64dd79fc10904fd06b3e9" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 42493 } } peers { permanent_uuid: "b856e73f795741b8a27cf02b90f8215f" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 44371 } } peers { permanent_uuid: "eaed6ab8f1a94a5b921e25fab99bd4f8" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 37023 } }
I20260928 21:58:06.499507 22410 leader_election.cc:288] T 0b34f03603394d99b0d8b9947b820ac5 P eaed6ab8f1a94a5b921e25fab99bd4f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 016cc66224c64dd79fc10904fd06b3e9 (127.2.182.67:42493), b856e73f795741b8a27cf02b90f8215f (127.2.182.66:44371)
I20260928 21:58:06.502421 22411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:06.499691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.67:42493 (local address 127.2.182.65:49407)
0928 21:58:06.499881 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:06.499891 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:06.499958 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:06.500094 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:06.500095 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:06.500097 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:58:06.500170 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:06.500172 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:06.500797 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:06.500798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:06.501615 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:06.501617 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:06.502174 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:06.502175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:06.502239 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:06.502242 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:58:06.502286 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 21:58:06.502329 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":11,"thread_start_us":78,"threads_started":1}
I20260928 21:58:06.502409 22398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:06.499808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.65:49407 (local address 127.2.182.67:42493)
0928 21:58:06.499910 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:06.499911 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:06.499944 (+ 33us) server_negotiation.cc:407] Connection header received
0928 21:58:06.500023 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:06.500024 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:06.500033 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:06.500077 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:58:06.500228 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:06.500717 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:06.501719 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:06.502088 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:06.502157 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:06.502166 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:06.502299 (+ 133us) server_negotiation.cc:299] Negotiation successful
0928 21:58:06.502345 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 21:58:06.502655 22412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:06.499739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.66:44371 (local address 127.2.182.65:60229)
0928 21:58:06.499900 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:06.499909 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:06.499966 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:06.500102 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:06.500103 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:06.500104 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:58:06.500185 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:06.500187 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:06.500792 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:06.500793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:06.501735 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:06.501739 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:06.502357 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:06.502358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:06.502432 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:06.502435 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:58:06.502485 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 21:58:06.502528 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":42,"spinlock_wait_cycles":12672,"thread_start_us":71,"threads_started":1}
I20260928 21:58:06.502799 22401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:06.499818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.65:60229 (local address 127.2.182.66:44371)
0928 21:58:06.499877 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:06.499880 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:06.499968 (+ 88us) server_negotiation.cc:407] Connection header received
0928 21:58:06.500036 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:06.500037 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:06.500045 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:06.500092 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:58:06.500242 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:06.500717 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:06.501846 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:06.502278 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:06.502345 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:06.502354 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:06.502654 (+ 300us) server_negotiation.cc:299] Negotiation successful
0928 21:58:06.502722 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 21:58:06.502919 22339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b34f03603394d99b0d8b9947b820ac5" candidate_uuid: "eaed6ab8f1a94a5b921e25fab99bd4f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "016cc66224c64dd79fc10904fd06b3e9" is_pre_election: true
I20260928 21:58:06.503082 22339 raft_consensus.cc:2454] T 0b34f03603394d99b0d8b9947b820ac5 P 016cc66224c64dd79fc10904fd06b3e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaed6ab8f1a94a5b921e25fab99bd4f8 in term 0.
I20260928 21:58:06.503119 22205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b34f03603394d99b0d8b9947b820ac5" candidate_uuid: "eaed6ab8f1a94a5b921e25fab99bd4f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b856e73f795741b8a27cf02b90f8215f" is_pre_election: true
I20260928 21:58:06.503289 22205 raft_consensus.cc:2454] T 0b34f03603394d99b0d8b9947b820ac5 P b856e73f795741b8a27cf02b90f8215f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaed6ab8f1a94a5b921e25fab99bd4f8 in term 0.
I20260928 21:58:06.503319 22004 leader_election.cc:302] T 0b34f03603394d99b0d8b9947b820ac5 P eaed6ab8f1a94a5b921e25fab99bd4f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 016cc66224c64dd79fc10904fd06b3e9, eaed6ab8f1a94a5b921e25fab99bd4f8; no voters:
I20260928 21:58:06.503476 22410 raft_consensus.cc:2781] T 0b34f03603394d99b0d8b9947b820ac5 P eaed6ab8f1a94a5b921e25fab99bd4f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:58:06.503551 22410 raft_consensus.cc:482] T 0b34f03603394d99b0d8b9947b820ac5 P eaed6ab8f1a94a5b921e25fab99bd4f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:58:06.503580 22410 raft_consensus.cc:3037] T 0b34f03603394d99b0d8b9947b820ac5 P eaed6ab8f1a94a5b921e25fab99bd4f8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:58:06.504271 22410 raft_consensus.cc:504] T 0b34f03603394d99b0d8b9947b820ac5 P eaed6ab8f1a94a5b921e25fab99bd4f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "016cc66224c64dd79fc10904fd06b3e9" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 42493 } } peers { permanent_uuid: "b856e73f795741b8a27cf02b90f8215f" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 44371 } } peers { permanent_uuid: "eaed6ab8f1a94a5b921e25fab99bd4f8" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 37023 } }
I20260928 21:58:06.504403 22410 leader_election.cc:288] T 0b34f03603394d99b0d8b9947b820ac5 P eaed6ab8f1a94a5b921e25fab99bd4f8 [CANDIDATE]: Term 1 election: Requested vote from peers 016cc66224c64dd79fc10904fd06b3e9 (127.2.182.67:42493), b856e73f795741b8a27cf02b90f8215f (127.2.182.66:44371)
I20260928 21:58:06.504612 22339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b34f03603394d99b0d8b9947b820ac5" candidate_uuid: "eaed6ab8f1a94a5b921e25fab99bd4f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "016cc66224c64dd79fc10904fd06b3e9"
I20260928 21:58:06.504701 22339 raft_consensus.cc:3037] T 0b34f03603394d99b0d8b9947b820ac5 P 016cc66224c64dd79fc10904fd06b3e9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:58:06.505399 22339 raft_consensus.cc:2454] T 0b34f03603394d99b0d8b9947b820ac5 P 016cc66224c64dd79fc10904fd06b3e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaed6ab8f1a94a5b921e25fab99bd4f8 in term 1.
I20260928 21:58:06.505511 22205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b34f03603394d99b0d8b9947b820ac5" candidate_uuid: "eaed6ab8f1a94a5b921e25fab99bd4f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b856e73f795741b8a27cf02b90f8215f"
I20260928 21:58:06.505733 22004 leader_election.cc:302] T 0b34f03603394d99b0d8b9947b820ac5 P eaed6ab8f1a94a5b921e25fab99bd4f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 016cc66224c64dd79fc10904fd06b3e9, eaed6ab8f1a94a5b921e25fab99bd4f8; no voters:
I20260928 21:58:06.505760 22205 raft_consensus.cc:3037] T 0b34f03603394d99b0d8b9947b820ac5 P b856e73f795741b8a27cf02b90f8215f [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:58:06.505832 22410 raft_consensus.cc:2781] T 0b34f03603394d99b0d8b9947b820ac5 P eaed6ab8f1a94a5b921e25fab99bd4f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:58:06.506024 22410 raft_consensus.cc:686] T 0b34f03603394d99b0d8b9947b820ac5 P eaed6ab8f1a94a5b921e25fab99bd4f8 [term 1 LEADER]: Becoming Leader. State: Replica: eaed6ab8f1a94a5b921e25fab99bd4f8, State: Running, Role: LEADER
I20260928 21:58:06.506246 22410 consensus_queue.cc:237] T 0b34f03603394d99b0d8b9947b820ac5 P eaed6ab8f1a94a5b921e25fab99bd4f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "016cc66224c64dd79fc10904fd06b3e9" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 42493 } } peers { permanent_uuid: "b856e73f795741b8a27cf02b90f8215f" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 44371 } } peers { permanent_uuid: "eaed6ab8f1a94a5b921e25fab99bd4f8" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 37023 } }
I20260928 21:58:06.506744 22205 raft_consensus.cc:2454] T 0b34f03603394d99b0d8b9947b820ac5 P b856e73f795741b8a27cf02b90f8215f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaed6ab8f1a94a5b921e25fab99bd4f8 in term 1.
I20260928 21:58:06.507068 21926 catalog_manager.cc:5495] T 0b34f03603394d99b0d8b9947b820ac5 P eaed6ab8f1a94a5b921e25fab99bd4f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to eaed6ab8f1a94a5b921e25fab99bd4f8 (127.2.182.65). New cstate: current_term: 1 leader_uuid: "eaed6ab8f1a94a5b921e25fab99bd4f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "016cc66224c64dd79fc10904fd06b3e9" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 42493 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b856e73f795741b8a27cf02b90f8215f" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 44371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaed6ab8f1a94a5b921e25fab99bd4f8" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 37023 } health_report { overall_health: HEALTHY } } }
W20260928 21:58:06.522989 22386 tablet.cc:2369] T 0b34f03603394d99b0d8b9947b820ac5 P 016cc66224c64dd79fc10904fd06b3e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:58:06.528095 22400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:06.525773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57974 (local address 127.2.182.65:37023)
0928 21:58:06.525833 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:06.525836 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:06.525855 (+ 19us) server_negotiation.cc:407] Connection header received
0928 21:58:06.525898 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:06.525899 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:06.525921 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:06.525974 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:58:06.526115 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:06.526569 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:06.527364 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:06.527559 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:06.527617 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:06.527757 (+ 140us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:58:06.527862 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:58:06.527932 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:06.527970 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 21:58:06.527991 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 21:58:06.529915 22339 raft_consensus.cc:1261] T 0b34f03603394d99b0d8b9947b820ac5 P 016cc66224c64dd79fc10904fd06b3e9 [term 1 FOLLOWER]: Refusing update from remote peer eaed6ab8f1a94a5b921e25fab99bd4f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:58:06.529915 22205 raft_consensus.cc:1261] T 0b34f03603394d99b0d8b9947b820ac5 P b856e73f795741b8a27cf02b90f8215f [term 1 FOLLOWER]: Refusing update from remote peer eaed6ab8f1a94a5b921e25fab99bd4f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:58:06.530287 22413 consensus_queue.cc:1035] T 0b34f03603394d99b0d8b9947b820ac5 P eaed6ab8f1a94a5b921e25fab99bd4f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b856e73f795741b8a27cf02b90f8215f" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 44371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:58:06.530433 22410 consensus_queue.cc:1035] T 0b34f03603394d99b0d8b9947b820ac5 P eaed6ab8f1a94a5b921e25fab99bd4f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "016cc66224c64dd79fc10904fd06b3e9" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 42493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:58:06.532613 22416 mvcc.cc:205] Tried to move back new op lower bound from 7334431484023791616 to 7334431483930304512. Current Snapshot: MvccSnapshot[applied={T|T < 7334431484023791616}]
I20260928 21:58:06.533598 22418 mvcc.cc:205] Tried to move back new op lower bound from 7334431484023791616 to 7334431483930304512. Current Snapshot: MvccSnapshot[applied={T|T < 7334431484023791616}]
I20260928 21:58:06.534936 21926 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49468:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:58:06.535041 21926 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49468:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:58:06.535823 21926 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 37a27d7dd1cd49038bb296c2bb86f989: Sending DeleteTablet for 3 replicas of tablet 0b34f03603394d99b0d8b9947b820ac5
I20260928 21:58:06.536262 22051 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b34f03603394d99b0d8b9947b820ac5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:58:06 UTC) from {username='kudu', principal='kudu/127.2.182.126@KRBTEST.COM'} at 127.0.0.1:57966
I20260928 21:58:06.536290 22319 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b34f03603394d99b0d8b9947b820ac5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:58:06 UTC) from {username='kudu', principal='kudu/127.2.182.126@KRBTEST.COM'} at 127.0.0.1:49268
I20260928 21:58:06.536619 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 21988
I20260928 21:58:06.538420 22185 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b34f03603394d99b0d8b9947b820ac5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:58:06 UTC) from {username='kudu', principal='kudu/127.2.182.126@KRBTEST.COM'} at 127.0.0.1:55340
I20260928 21:58:06.538656 22428 tablet_replica.cc:330] T 0b34f03603394d99b0d8b9947b820ac5 P b856e73f795741b8a27cf02b90f8215f: stopping tablet replica
I20260928 21:58:06.538669 22427 tablet_replica.cc:330] T 0b34f03603394d99b0d8b9947b820ac5 P 016cc66224c64dd79fc10904fd06b3e9: stopping tablet replica
I20260928 21:58:06.538774 22427 raft_consensus.cc:2229] T 0b34f03603394d99b0d8b9947b820ac5 P 016cc66224c64dd79fc10904fd06b3e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:58:06.538784 22428 raft_consensus.cc:2229] T 0b34f03603394d99b0d8b9947b820ac5 P b856e73f795741b8a27cf02b90f8215f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:58:06.538911 22428 raft_consensus.cc:2258] T 0b34f03603394d99b0d8b9947b820ac5 P b856e73f795741b8a27cf02b90f8215f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:58:06.538915 22427 raft_consensus.cc:2258] T 0b34f03603394d99b0d8b9947b820ac5 P 016cc66224c64dd79fc10904fd06b3e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:58:06.539371 22428 ts_tablet_manager.cc:1905] T 0b34f03603394d99b0d8b9947b820ac5 P b856e73f795741b8a27cf02b90f8215f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:58:06.539376 22427 ts_tablet_manager.cc:1905] T 0b34f03603394d99b0d8b9947b820ac5 P 016cc66224c64dd79fc10904fd06b3e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:58:06.540925 22427 ts_tablet_manager.cc:1918] T 0b34f03603394d99b0d8b9947b820ac5 P 016cc66224c64dd79fc10904fd06b3e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:58:06.540920 22428 ts_tablet_manager.cc:1918] T 0b34f03603394d99b0d8b9947b820ac5 P b856e73f795741b8a27cf02b90f8215f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:58:06.540992 22427 log.cc:1196] T 0b34f03603394d99b0d8b9947b820ac5 P 016cc66224c64dd79fc10904fd06b3e9: Deleting WAL directory at /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/minicluster-data/ts-2/wal/wals/0b34f03603394d99b0d8b9947b820ac5
I20260928 21:58:06.540992 22428 log.cc:1196] T 0b34f03603394d99b0d8b9947b820ac5 P b856e73f795741b8a27cf02b90f8215f: Deleting WAL directory at /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632440055392-2777-0/minicluster-data/ts-1/wal/wals/0b34f03603394d99b0d8b9947b820ac5
I20260928 21:58:06.541254 22428 ts_tablet_manager.cc:1939] T 0b34f03603394d99b0d8b9947b820ac5 P b856e73f795741b8a27cf02b90f8215f: Deleting consensus metadata
I20260928 21:58:06.541275 22427 ts_tablet_manager.cc:1939] T 0b34f03603394d99b0d8b9947b820ac5 P 016cc66224c64dd79fc10904fd06b3e9: Deleting consensus metadata
I20260928 21:58:06.541734 21910 catalog_manager.cc:4841] TS 016cc66224c64dd79fc10904fd06b3e9 (127.2.182.67:42493): tablet 0b34f03603394d99b0d8b9947b820ac5 (table test-table [id=c758c1a65d104f779fbc1705a7e0524d]) successfully deleted
I20260928 21:58:06.541823 21912 catalog_manager.cc:4841] TS b856e73f795741b8a27cf02b90f8215f (127.2.182.66:44371): tablet 0b34f03603394d99b0d8b9947b820ac5 (table test-table [id=c758c1a65d104f779fbc1705a7e0524d]) successfully deleted
W20260928 21:58:06.542409 21910 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.182.65:37023 (error 108)
I20260928 21:58:06.542654 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 22122
I20260928 21:58:06.543187 22397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:06.542804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.65:37023 (local address 127.0.0.1:57980)
0928 21:58:06.543122 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:06.543160 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.65:37023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":299}
W20260928 21:58:06.543432 21910 catalog_manager.cc:4584] TS eaed6ab8f1a94a5b921e25fab99bd4f8 (127.2.182.65:37023): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0b34f03603394d99b0d8b9947b820ac5: Network error: Client connection negotiation failed: client connection to 127.2.182.65:37023: connect: Connection refused (error 111)
I20260928 21:58:06.547492 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 22256
I20260928 21:58:06.551816 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 21894
2026-09-28T21:58:06Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1995 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-28T21:58:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:58:06Z Disabled control of system clock
I20260928 21:58:06.577488 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632440055392-2777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632440055392-2777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632440055392-2777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:36291
--webserver_interface=127.2.182.126
--webserver_port=0
--builtin_ntp_servers=127.2.182.84:45297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.126:36291
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 21:58:06.658038 22433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:58:06.658295 22433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:58:06.658326 22433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:58:06.659564 22433 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:58:06.659610 22433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:58:06.659622 22433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:58:06.659633 22433 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:58:06.659643 22433 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:58:06.661005 22433 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:45297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632440055392-2777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632440055392-2777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.126:36291
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:36291
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632440055392-2777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:58:06.661247 22433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:58:06.661540 22433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:58:06.666256 22439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:06.666314 22438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:06.666374 22441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:06.666544 22433 server_base.cc:1034] running on GCE node
I20260928 21:58:06.666805 22433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:58:06.667101 22433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:58:06.668244 22433 hybrid_clock.cc:648] HybridClock initialized: now 1790632686668235 us; error 33 us; skew 500 ppm
I20260928 21:58:06.669588 22433 webserver.cc:460] Webserver started at http://127.2.182.126:33413/ using document root <none> and password file <none>
I20260928 21:58:06.669809 22433 fs_manager.cc:359] Metadata directory not provided
I20260928 21:58:06.669857 22433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:58:06.669996 22433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:58:06.671072 22433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632440055392-2777-0/minicluster-data/master-0/data/instance:
uuid: "fe81677b40c04463b85b9ff048426c75"
format_stamp: "Formatted at 2026-09-28 21:58:06 on dist-test-slave-k6lz"
server_key: "6c1bdc7f2a24ad7f3a721680104d55ff"
server_key_iv: "c9cef19f17b9c4d86dfebf2620a06eb3"
server_key_version: "encryptionkey@0"
I20260928 21:58:06.671458 22433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632440055392-2777-0/minicluster-data/master-0/wal/instance:
uuid: "fe81677b40c04463b85b9ff048426c75"
format_stamp: "Formatted at 2026-09-28 21:58:06 on dist-test-slave-k6lz"
server_key: "6c1bdc7f2a24ad7f3a721680104d55ff"
server_key_iv: "c9cef19f17b9c4d86dfebf2620a06eb3"
server_key_version: "encryptionkey@0"
I20260928 21:58:06.672660 22433 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:58:06.673369 22447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:06.673524 22433 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 21:58:06.673617 22433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632440055392-2777-0/minicluster-data/master-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632440055392-2777-0/minicluster-data/master-0/wal
uuid: "fe81677b40c04463b85b9ff048426c75"
format_stamp: "Formatted at 2026-09-28 21:58:06 on dist-test-slave-k6lz"
server_key: "6c1bdc7f2a24ad7f3a721680104d55ff"
server_key_iv: "c9cef19f17b9c4d86dfebf2620a06eb3"
server_key_version: "encryptionkey@0"
I20260928 21:58:06.673687 22433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632440055392-2777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632440055392-2777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632440055392-2777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:58:06.683382 22433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:58:06.683691 22433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:58:06.683818 22433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:58:06.687718 22433 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.126:36291
I20260928 21:58:06.687750 22499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.126:36291 every 8 connection(s)
I20260928 21:58:06.688074 22433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
I20260928 21:58:06.689100 22500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:58:06.691452 22500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe81677b40c04463b85b9ff048426c75: Bootstrap starting.
I20260928 21:58:06.691651 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 22433
I20260928 21:58:06.691738 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632440055392-2777-0/minicluster-data/master-0/wal/instance
I20260928 21:58:06.691915 2777 external_mini_cluster.cc:1428] Setting key 4631f655000e875510583caa3a677fd5
I20260928 21:58:06.692083 22500 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe81677b40c04463b85b9ff048426c75: Neither blocks nor log segments found. Creating new log.
I20260928 21:58:06.692375 22500 log.cc:824] T 00000000000000000000000000000000 P fe81677b40c04463b85b9ff048426c75: Log is configured to *not* fsync() on all Append() calls
I20260928 21:58:06.693046 22500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe81677b40c04463b85b9ff048426c75: No bootstrap required, opened a new log
I20260928 21:58:06.694942 22500 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe81677b40c04463b85b9ff048426c75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe81677b40c04463b85b9ff048426c75" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 36291 } }
I20260928 21:58:06.695081 22500 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe81677b40c04463b85b9ff048426c75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:58:06.695112 22500 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe81677b40c04463b85b9ff048426c75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe81677b40c04463b85b9ff048426c75, State: Initialized, Role: FOLLOWER
I20260928 21:58:06.695261 22500 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe81677b40c04463b85b9ff048426c75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe81677b40c04463b85b9ff048426c75" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 36291 } }
I20260928 21:58:06.695365 22500 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe81677b40c04463b85b9ff048426c75 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:58:06.695411 22500 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe81677b40c04463b85b9ff048426c75 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:58:06.695454 22500 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe81677b40c04463b85b9ff048426c75 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:58:06.696157 22500 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe81677b40c04463b85b9ff048426c75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe81677b40c04463b85b9ff048426c75" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 36291 } }
I20260928 21:58:06.696261 22500 leader_election.cc:302] T 00000000000000000000000000000000 P fe81677b40c04463b85b9ff048426c75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe81677b40c04463b85b9ff048426c75; no voters:
I20260928 21:58:06.696436 22500 leader_election.cc:288] T 00000000000000000000000000000000 P fe81677b40c04463b85b9ff048426c75 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:58:06.696483 22505 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe81677b40c04463b85b9ff048426c75 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:58:06.696589 22505 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe81677b40c04463b85b9ff048426c75 [term 1 LEADER]: Becoming Leader. State: Replica: fe81677b40c04463b85b9ff048426c75, State: Running, Role: LEADER
I20260928 21:58:06.696662 22505 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe81677b40c04463b85b9ff048426c75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe81677b40c04463b85b9ff048426c75" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 36291 } }
I20260928 21:58:06.696750 22500 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe81677b40c04463b85b9ff048426c75 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:58:06.696964 22505 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe81677b40c04463b85b9ff048426c75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe81677b40c04463b85b9ff048426c75. Latest consensus state: current_term: 1 leader_uuid: "fe81677b40c04463b85b9ff048426c75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe81677b40c04463b85b9ff048426c75" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 36291 } } }
I20260928 21:58:06.697026 22505 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe81677b40c04463b85b9ff048426c75 [sys.catalog]: This master's current role is: LEADER
I20260928 21:58:06.697347 22503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:06.692395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49312 (local address 127.2.182.126:36291)
0928 21:58:06.692615 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:06.692624 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:06.692656 (+ 32us) server_negotiation.cc:407] Connection header received
0928 21:58:06.693150 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:06.693160 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:06.693229 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:06.693359 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:06.693572 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:06.694464 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:06.695200 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:06.695811 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:06.695963 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:06.696572 (+ 609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:06.696576 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:06.696577 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:58:06.696785 (+ 208us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:06.696826 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:58:06.696839 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:06.696892 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:06.697067 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 21:58:06.697149 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":74,"threads_started":1}
I20260928 21:58:06.697638 22507 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe81677b40c04463b85b9ff048426c75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe81677b40c04463b85b9ff048426c75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe81677b40c04463b85b9ff048426c75" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 36291 } } }
I20260928 21:58:06.697713 22507 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe81677b40c04463b85b9ff048426c75 [sys.catalog]: This master's current role is: LEADER
I20260928 21:58:06.697892 22509 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:58:06.698333 22509 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:58:06.699767 22509 catalog_manager.cc:1347] Generated new cluster ID: 8870f2089a164a1086ae9048821bdc48
I20260928 21:58:06.699808 22509 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:58:06.708253 22509 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:58:06.708803 22509 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:58:06.717935 22509 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe81677b40c04463b85b9ff048426c75: Generated new TSK 0
I20260928 21:58:06.718128 22509 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:58:06.722851 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632440055392-2777-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632440055392-2777-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632440055392-2777-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.65:0
--local_ip_for_outbound_sockets=127.2.182.65
--webserver_interface=127.2.182.65
--webserver_port=0
--tserver_master_addrs=127.2.182.126:36291
--builtin_ntp_servers=127.2.182.84:45297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 21:58:06.803706 22524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:58:06.803921 22524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:58:06.803951 22524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:58:06.805107 22524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:58:06.805147 22524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:58:06.805181 22524 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.65
I20260928 21:58:06.806505 22524 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:45297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632440055392-2777-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632440055392-2777-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.182.65
--webserver_port=0
--tserver_master_addrs=127.2.182.126:36291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22524
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.65
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632440055392-2777-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:58:06.806751 22524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:58:06.807031 22524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:58:06.811975 22529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:06.812083 22532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:06.812229 22524 server_base.cc:1034] running on GCE node
W20260928 21:58:06.811975 22530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:06.812616 22524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:58:06.812930 22524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:58:06.814127 22524 hybrid_clock.cc:648] HybridClock initialized: now 1790632686814110 us; error 33 us; skew 500 ppm
I20260928 21:58:06.815730 22524 webserver.cc:460] Webserver started at http://127.2.182.65:42659/ using document root <none> and password file <none>
I20260928 21:58:06.816046 22524 fs_manager.cc:359] Metadata directory not provided
I20260928 21:58:06.816113 22524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:58:06.816252 22524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:58:06.817660 22524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632440055392-2777-0/minicluster-data/ts-0/data/instance:
uuid: "4cfba59d25694d2e8403bb8980334d9b"
format_stamp: "Formatted at 2026-09-28 21:58:06 on dist-test-slave-k6lz"
server_key: "cc54529c87ddea423df4aa330dc2f924"
server_key_iv: "260693c688612f732570f995cfefb194"
server_key_version: "encryptionkey@0"
I20260928 21:58:06.818154 22524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632440055392-2777-0/minicluster-data/ts-0/wal/instance:
uuid: "4cfba59d25694d2e8403bb8980334d9b"
format_stamp: "Formatted at 2026-09-28 21:58:06 on dist-test-slave-k6lz"
server_key: "cc54529c87ddea423df4aa330dc2f924"
server_key_iv: "260693c688612f732570f995cfefb194"
server_key_version: "encryptionkey@0"
I20260928 21:58:06.819746 22524 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260928 21:58:06.820675 22538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:06.820864 22524 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:58:06.820951 22524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632440055392-2777-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632440055392-2777-0/minicluster-data/ts-0/wal
uuid: "4cfba59d25694d2e8403bb8980334d9b"
format_stamp: "Formatted at 2026-09-28 21:58:06 on dist-test-slave-k6lz"
server_key: "cc54529c87ddea423df4aa330dc2f924"
server_key_iv: "260693c688612f732570f995cfefb194"
server_key_version: "encryptionkey@0"
I20260928 21:58:06.821023 22524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632440055392-2777-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632440055392-2777-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632440055392-2777-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:58:06.841837 22524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:58:06.842142 22524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:58:06.842248 22524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:58:06.842448 22524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:58:06.842792 22524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:58:06.842829 22524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:06.842852 22524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:58:06.842864 22524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:06.848735 22524 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.65:36959
I20260928 21:58:06.848776 22651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.65:36959 every 8 connection(s)
I20260928 21:58:06.849098 22524 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632440055392-2777-0/minicluster-data/ts-0/data/info.pb
I20260928 21:58:06.854183 22503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:06.849877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.65:52509 (local address 127.2.182.126:36291)
0928 21:58:06.849968 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:06.849970 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:06.850156 (+ 186us) server_negotiation.cc:407] Connection header received
0928 21:58:06.850633 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:06.850634 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:06.850645 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:06.850708 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:06.851450 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:06.851949 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:06.852920 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:06.853102 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:06.853152 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:06.853916 (+ 764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:06.853917 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:06.853917 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:58:06.853921 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:06.853935 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:58:06.853939 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:06.853982 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:06.854099 (+ 117us) server_negotiation.cc:299] Negotiation successful
0928 21:58:06.854121 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 21:58:06.854411 22654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:06.849845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:36291 (local address 127.2.182.65:52509)
0928 21:58:06.850038 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:06.850079 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:06.850546 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:06.850768 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:06.850771 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:06.850780 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:58:06.851360 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:06.851366 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:06.852047 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:06.852048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:06.852828 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:06.852829 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:06.853172 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:06.853172 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:06.853283 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:06.853776 (+ 493us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:58:06.853777 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:58:06.853806 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:58:06.853828 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:58:06.854017 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:58:06.854024 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:58:06.854026 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 21:58:06.854095 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 21:58:06.854182 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":83,"threads_started":1}
I20260928 21:58:06.854991 22652 heartbeater.cc:347] Connected to a master server at 127.2.182.126:36291
I20260928 21:58:06.855087 22652 heartbeater.cc:464] Registering TS with master...
I20260928 21:58:06.855262 22652 heartbeater.cc:511] Master 127.2.182.126:36291 requested a full tablet report, sending...
I20260928 21:58:06.855780 22464 ts_manager.cc:194] Registered new tserver with Master: 4cfba59d25694d2e8403bb8980334d9b (127.2.182.65:36959)
I20260928 21:58:06.856487 22464 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.182.65:52509
I20260928 21:58:06.858104 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 22524
I20260928 21:58:06.858181 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632440055392-2777-0/minicluster-data/ts-0/wal/instance
I20260928 21:58:06.858352 2777 external_mini_cluster.cc:1428] Setting key e67e78b6adf7c06817de801927e8d30e
I20260928 21:58:06.859236 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632440055392-2777-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632440055392-2777-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632440055392-2777-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.66:0
--local_ip_for_outbound_sockets=127.2.182.66
--webserver_interface=127.2.182.66
--webserver_port=0
--tserver_master_addrs=127.2.182.126:36291
--builtin_ntp_servers=127.2.182.84:45297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 21:58:06.938874 22655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:58:06.939098 22655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:58:06.939126 22655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:58:06.940291 22655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:58:06.940332 22655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:58:06.940366 22655 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.66
I20260928 21:58:06.941764 22655 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:45297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632440055392-2777-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632440055392-2777-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.182.66
--webserver_port=0
--tserver_master_addrs=127.2.182.126:36291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.66
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632440055392-2777-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:58:06.941988 22655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:58:06.942250 22655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:58:06.947016 22661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:06.947016 22663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:06.947198 22655 server_base.cc:1034] running on GCE node
W20260928 21:58:06.947094 22660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:06.947464 22655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:58:06.947665 22655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:58:06.948799 22655 hybrid_clock.cc:648] HybridClock initialized: now 1790632686948783 us; error 32 us; skew 500 ppm
I20260928 21:58:06.949975 22655 webserver.cc:460] Webserver started at http://127.2.182.66:41089/ using document root <none> and password file <none>
I20260928 21:58:06.950153 22655 fs_manager.cc:359] Metadata directory not provided
I20260928 21:58:06.950192 22655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:58:06.950280 22655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:58:06.951277 22655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632440055392-2777-0/minicluster-data/ts-1/data/instance:
uuid: "42f4ffc7e1c743ecae94261eca355f2f"
format_stamp: "Formatted at 2026-09-28 21:58:06 on dist-test-slave-k6lz"
server_key: "85ff565e3b8748c847cd4761b47139c6"
server_key_iv: "4741faad1c9d19aed047edc65e6a4076"
server_key_version: "encryptionkey@0"
I20260928 21:58:06.951696 22655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632440055392-2777-0/minicluster-data/ts-1/wal/instance:
uuid: "42f4ffc7e1c743ecae94261eca355f2f"
format_stamp: "Formatted at 2026-09-28 21:58:06 on dist-test-slave-k6lz"
server_key: "85ff565e3b8748c847cd4761b47139c6"
server_key_iv: "4741faad1c9d19aed047edc65e6a4076"
server_key_version: "encryptionkey@0"
I20260928 21:58:06.952809 22655 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:58:06.953441 22669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:06.953660 22655 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:58:06.953745 22655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632440055392-2777-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632440055392-2777-0/minicluster-data/ts-1/wal
uuid: "42f4ffc7e1c743ecae94261eca355f2f"
format_stamp: "Formatted at 2026-09-28 21:58:06 on dist-test-slave-k6lz"
server_key: "85ff565e3b8748c847cd4761b47139c6"
server_key_iv: "4741faad1c9d19aed047edc65e6a4076"
server_key_version: "encryptionkey@0"
I20260928 21:58:06.953799 22655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632440055392-2777-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632440055392-2777-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632440055392-2777-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:58:06.969491 22655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:58:06.969754 22655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:58:06.969856 22655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:58:06.970042 22655 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:58:06.970312 22655 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:58:06.970341 22655 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:06.970362 22655 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:58:06.970373 22655 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:06.975711 22655 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.66:35239
I20260928 21:58:06.975751 22782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.66:35239 every 8 connection(s)
I20260928 21:58:06.976059 22655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632440055392-2777-0/minicluster-data/ts-1/data/info.pb
I20260928 21:58:06.981356 22503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:06.976740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.66:54695 (local address 127.2.182.126:36291)
0928 21:58:06.976802 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:06.976804 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:06.977068 (+ 264us) server_negotiation.cc:407] Connection header received
0928 21:58:06.977591 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:06.977592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:06.977602 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:06.977652 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:06.978350 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:06.978930 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:06.980042 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:06.980232 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:06.980291 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:06.981091 (+ 800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:06.981093 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:06.981094 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:58:06.981098 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:06.981112 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:58:06.981116 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:06.981161 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:06.981272 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 21:58:06.981294 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 21:58:06.981644 22785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:06.976721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:36291 (local address 127.2.182.66:54695)
0928 21:58:06.976953 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:06.976989 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:06.977506 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:06.977706 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:06.977709 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:06.977718 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:58:06.978226 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:06.978231 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:06.979036 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:06.979037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:06.979957 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:06.979960 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:06.980301 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:06.980302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:06.980427 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:06.980940 (+ 513us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:58:06.980940 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:58:06.980969 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:58:06.980994 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:58:06.981199 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:58:06.981205 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:58:06.981207 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 21:58:06.981266 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 21:58:06.981352 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":88,"threads_started":1}
I20260928 21:58:06.982169 22783 heartbeater.cc:347] Connected to a master server at 127.2.182.126:36291
I20260928 21:58:06.982259 22783 heartbeater.cc:464] Registering TS with master...
I20260928 21:58:06.982443 22783 heartbeater.cc:511] Master 127.2.182.126:36291 requested a full tablet report, sending...
I20260928 21:58:06.982807 22464 ts_manager.cc:194] Registered new tserver with Master: 42f4ffc7e1c743ecae94261eca355f2f (127.2.182.66:35239)
I20260928 21:58:06.983183 22464 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.182.66:54695
I20260928 21:58:06.983840 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 22655
I20260928 21:58:06.983928 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632440055392-2777-0/minicluster-data/ts-1/wal/instance
I20260928 21:58:06.984082 2777 external_mini_cluster.cc:1428] Setting key afd57c7411ad62e26de76d4b9e5b13ec
I20260928 21:58:06.984964 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632440055392-2777-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632440055392-2777-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632440055392-2777-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.67:0
--local_ip_for_outbound_sockets=127.2.182.67
--webserver_interface=127.2.182.67
--webserver_port=0
--tserver_master_addrs=127.2.182.126:36291
--builtin_ntp_servers=127.2.182.84:45297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 21:58:07.064235 22786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:58:07.064440 22786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:58:07.064460 22786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:58:07.065586 22786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:58:07.065618 22786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:58:07.065641 22786 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.67
I20260928 21:58:07.066946 22786 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:45297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632440055392-2777-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632440055392-2777-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.182.67
--webserver_port=0
--tserver_master_addrs=127.2.182.126:36291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.67
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632440055392-2777-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:58:07.067150 22786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:58:07.067428 22786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:58:07.072333 22794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:07.072319 22791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:07.072446 22786 server_base.cc:1034] running on GCE node
W20260928 21:58:07.072314 22792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:07.072714 22786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:58:07.072976 22786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:58:07.074141 22786 hybrid_clock.cc:648] HybridClock initialized: now 1790632687074130 us; error 31 us; skew 500 ppm
I20260928 21:58:07.075373 22786 webserver.cc:460] Webserver started at http://127.2.182.67:33377/ using document root <none> and password file <none>
I20260928 21:58:07.075611 22786 fs_manager.cc:359] Metadata directory not provided
I20260928 21:58:07.075662 22786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:58:07.075786 22786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:58:07.076870 22786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632440055392-2777-0/minicluster-data/ts-2/data/instance:
uuid: "e6fa6593e89c432084ef4414a59b528d"
format_stamp: "Formatted at 2026-09-28 21:58:07 on dist-test-slave-k6lz"
server_key: "982df9201af62438eeaf011755d2d15f"
server_key_iv: "7b2b2352c5f6db0e2e2023b7b619a4a9"
server_key_version: "encryptionkey@0"
I20260928 21:58:07.077191 22786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632440055392-2777-0/minicluster-data/ts-2/wal/instance:
uuid: "e6fa6593e89c432084ef4414a59b528d"
format_stamp: "Formatted at 2026-09-28 21:58:07 on dist-test-slave-k6lz"
server_key: "982df9201af62438eeaf011755d2d15f"
server_key_iv: "7b2b2352c5f6db0e2e2023b7b619a4a9"
server_key_version: "encryptionkey@0"
I20260928 21:58:07.078454 22786 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 21:58:07.079136 22800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:07.079317 22786 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 21:58:07.079412 22786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632440055392-2777-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632440055392-2777-0/minicluster-data/ts-2/wal
uuid: "e6fa6593e89c432084ef4414a59b528d"
format_stamp: "Formatted at 2026-09-28 21:58:07 on dist-test-slave-k6lz"
server_key: "982df9201af62438eeaf011755d2d15f"
server_key_iv: "7b2b2352c5f6db0e2e2023b7b619a4a9"
server_key_version: "encryptionkey@0"
I20260928 21:58:07.079465 22786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632440055392-2777-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632440055392-2777-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632440055392-2777-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:58:07.104250 22786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:58:07.104630 22786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:58:07.104739 22786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:58:07.104928 22786 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:58:07.105199 22786 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:58:07.105227 22786 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:07.105249 22786 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:58:07.105261 22786 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:07.110370 22786 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.67:37147
I20260928 21:58:07.110430 22913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.67:37147 every 8 connection(s)
I20260928 21:58:07.110716 22786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632440055392-2777-0/minicluster-data/ts-2/data/info.pb
I20260928 21:58:07.115734 22503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:07.111405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.67:38679 (local address 127.2.182.126:36291)
0928 21:58:07.111448 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:07.111450 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:07.111720 (+ 270us) server_negotiation.cc:407] Connection header received
0928 21:58:07.112227 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:07.112228 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:07.112241 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:07.112291 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:07.112947 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:07.113442 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:07.114376 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:07.114565 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:07.114619 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:07.115489 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:07.115490 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:07.115490 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:58:07.115495 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:07.115510 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:58:07.115514 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:07.115554 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:07.115629 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 21:58:07.115653 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 21:58:07.115947 22916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:07.111399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:36291 (local address 127.2.182.67:38679)
0928 21:58:07.111603 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:07.111637 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:07.112133 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:07.112358 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:07.112362 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:07.112373 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:58:07.112865 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:07.112869 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:07.113527 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:07.113528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:07.114294 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:07.114296 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:07.114638 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:07.114639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:07.114760 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:07.115319 (+ 559us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:58:07.115320 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:58:07.115378 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:58:07.115408 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:58:07.115565 (+ 157us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:58:07.115570 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:58:07.115573 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:58:07.115634 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 21:58:07.115739 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":64,"threads_started":1}
I20260928 21:58:07.116366 22914 heartbeater.cc:347] Connected to a master server at 127.2.182.126:36291
I20260928 21:58:07.116441 22914 heartbeater.cc:464] Registering TS with master...
I20260928 21:58:07.116622 22914 heartbeater.cc:511] Master 127.2.182.126:36291 requested a full tablet report, sending...
I20260928 21:58:07.117010 22464 ts_manager.cc:194] Registered new tserver with Master: e6fa6593e89c432084ef4414a59b528d (127.2.182.67:37147)
I20260928 21:58:07.117403 22464 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.182.67:38679
I20260928 21:58:07.119822 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 22786
I20260928 21:58:07.119911 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632440055392-2777-0/minicluster-data/ts-2/wal/instance
I20260928 21:58:07.120091 2777 external_mini_cluster.cc:1428] Setting key b207d30a30dc0e12c4852b3d7ff8fb75
I20260928 21:58:07.120930 2777 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:58:07.125061 22503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:07.122220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49314 (local address 127.2.182.126:36291)
0928 21:58:07.122257 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:07.122259 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:07.122358 (+ 99us) server_negotiation.cc:407] Connection header received
0928 21:58:07.122419 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:07.122420 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:07.122429 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:07.122473 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:07.122651 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:07.123126 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:07.123876 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:07.124093 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:07.124147 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:07.124791 (+ 644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:07.124793 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:07.124793 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:58:07.124798 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:07.124812 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:58:07.124816 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:07.124901 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:07.124950 (+ 49us) server_negotiation.cc:299] Negotiation successful
0928 21:58:07.124974 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 21:58:07.126634 22464 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49314:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:58:07.128216 22464 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:58:07.134469 22927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:07.130775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43432 (local address 127.2.182.66:35239)
0928 21:58:07.131113 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:07.131117 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:07.131126 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:58:07.131176 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:07.131178 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:07.131194 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:07.131262 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:58:07.131459 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:07.132139 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:07.133530 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:07.134071 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:07.134135 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:07.134229 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:07.134346 (+ 117us) server_negotiation.cc:299] Negotiation successful
0928 21:58:07.134384 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":85,"threads_started":1}
I20260928 21:58:07.134495 22923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:07.130548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.66:35239 (local address 127.0.0.1:43432)
0928 21:58:07.130888 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:07.130896 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:07.130931 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:07.131324 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:07.131325 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:07.131350 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:58:07.131393 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:07.131395 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:07.132221 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:07.132222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:07.133435 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:07.133438 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:07.134148 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:07.134149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:07.134225 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:07.134239 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 21:58:07.134302 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 21:58:07.134372 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":61,"threads_started":1}
I20260928 21:58:07.134514 22925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:07.130302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.67:37147 (local address 127.0.0.1:57448)
0928 21:58:07.130671 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:07.130711 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:07.130775 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:07.131032 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:07.131035 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:07.131037 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:58:07.131102 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:07.131104 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:07.132566 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:07.132567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:07.133415 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:07.133418 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:07.134246 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:07.134248 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:07.134308 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:07.134313 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 21:58:07.134358 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 21:58:07.134397 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":85,"threads_started":1}
I20260928 21:58:07.134610 22922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:07.130302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.65:36959 (local address 127.0.0.1:59284)
0928 21:58:07.130800 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:07.130817 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:07.130961 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:07.131197 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:07.131198 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:07.131200 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:58:07.131232 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:07.131233 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:07.132536 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:07.132537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:07.133580 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:07.133582 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:07.134433 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:07.134434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:07.134492 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:07.134496 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:58:07.134521 (+ 25us) client_negotiation.cc:240] Negotiation successful
0928 21:58:07.134554 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"mutex_wait_us":3,"spinlock_wait_cycles":384,"thread_start_us":87,"threads_started":1}
I20260928 21:58:07.134938 22926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:07.130592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57448 (local address 127.2.182.67:37147)
0928 21:58:07.130768 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:07.130773 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:07.130784 (+ 11us) server_negotiation.cc:407] Connection header received
0928 21:58:07.130862 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:07.130864 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:07.130879 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:07.130957 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:58:07.131181 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:07.132453 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:07.133495 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:07.134149 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:07.134687 (+ 538us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:07.134785 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:07.134812 (+ 27us) server_negotiation.cc:299] Negotiation successful
0928 21:58:07.134853 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":86,"threads_started":1}
I20260928 21:58:07.135210 22924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:07.130430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59284 (local address 127.2.182.65:36959)
0928 21:58:07.130657 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:07.130662 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:07.130927 (+ 265us) server_negotiation.cc:407] Connection header received
0928 21:58:07.131089 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:07.131092 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:07.131107 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:07.131293 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:58:07.131367 (+ 74us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:07.132458 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:07.133652 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:07.134352 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:07.134928 (+ 576us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:07.135086 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:07.135106 (+ 20us) server_negotiation.cc:299] Negotiation successful
0928 21:58:07.135135 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":104,"threads_started":1}
I20260928 21:58:07.135207 22717 tablet_service.cc:1467] Processing CreateTablet for tablet bc11e362a8474b87ae137bcc0bb563de (DEFAULT_TABLE table=test-table [id=a57bf8024e3d4410a41ee2a4ade5f4e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:58:07.135588 22717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc11e362a8474b87ae137bcc0bb563de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:58:07.135564 22848 tablet_service.cc:1467] Processing CreateTablet for tablet bc11e362a8474b87ae137bcc0bb563de (DEFAULT_TABLE table=test-table [id=a57bf8024e3d4410a41ee2a4ade5f4e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:58:07.135833 22848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc11e362a8474b87ae137bcc0bb563de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:58:07.138070 22929 tablet_bootstrap.cc:492] T bc11e362a8474b87ae137bcc0bb563de P e6fa6593e89c432084ef4414a59b528d: Bootstrap starting.
I20260928 21:58:07.138013 22928 tablet_bootstrap.cc:492] T bc11e362a8474b87ae137bcc0bb563de P 42f4ffc7e1c743ecae94261eca355f2f: Bootstrap starting.
I20260928 21:58:07.138686 22929 tablet_bootstrap.cc:654] T bc11e362a8474b87ae137bcc0bb563de P e6fa6593e89c432084ef4414a59b528d: Neither blocks nor log segments found. Creating new log.
I20260928 21:58:07.138671 22586 tablet_service.cc:1467] Processing CreateTablet for tablet bc11e362a8474b87ae137bcc0bb563de (DEFAULT_TABLE table=test-table [id=a57bf8024e3d4410a41ee2a4ade5f4e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:58:07.138895 22928 tablet_bootstrap.cc:654] T bc11e362a8474b87ae137bcc0bb563de P 42f4ffc7e1c743ecae94261eca355f2f: Neither blocks nor log segments found. Creating new log.
I20260928 21:58:07.138969 22929 log.cc:824] T bc11e362a8474b87ae137bcc0bb563de P e6fa6593e89c432084ef4414a59b528d: Log is configured to *not* fsync() on all Append() calls
I20260928 21:58:07.138998 22586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc11e362a8474b87ae137bcc0bb563de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:58:07.139182 22928 log.cc:824] T bc11e362a8474b87ae137bcc0bb563de P 42f4ffc7e1c743ecae94261eca355f2f: Log is configured to *not* fsync() on all Append() calls
I20260928 21:58:07.139683 22929 tablet_bootstrap.cc:492] T bc11e362a8474b87ae137bcc0bb563de P e6fa6593e89c432084ef4414a59b528d: No bootstrap required, opened a new log
I20260928 21:58:07.139742 22929 ts_tablet_manager.cc:1397] T bc11e362a8474b87ae137bcc0bb563de P e6fa6593e89c432084ef4414a59b528d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:58:07.139892 22928 tablet_bootstrap.cc:492] T bc11e362a8474b87ae137bcc0bb563de P 42f4ffc7e1c743ecae94261eca355f2f: No bootstrap required, opened a new log
I20260928 21:58:07.139958 22928 ts_tablet_manager.cc:1397] T bc11e362a8474b87ae137bcc0bb563de P 42f4ffc7e1c743ecae94261eca355f2f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 21:58:07.141659 22929 raft_consensus.cc:348] T bc11e362a8474b87ae137bcc0bb563de P e6fa6593e89c432084ef4414a59b528d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6fa6593e89c432084ef4414a59b528d" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 37147 } } peers { permanent_uuid: "4cfba59d25694d2e8403bb8980334d9b" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 36959 } } peers { permanent_uuid: "42f4ffc7e1c743ecae94261eca355f2f" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 35239 } }
I20260928 21:58:07.141781 22929 raft_consensus.cc:374] T bc11e362a8474b87ae137bcc0bb563de P e6fa6593e89c432084ef4414a59b528d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:58:07.141810 22929 raft_consensus.cc:729] T bc11e362a8474b87ae137bcc0bb563de P e6fa6593e89c432084ef4414a59b528d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6fa6593e89c432084ef4414a59b528d, State: Initialized, Role: FOLLOWER
I20260928 21:58:07.141938 22929 consensus_queue.cc:260] T bc11e362a8474b87ae137bcc0bb563de P e6fa6593e89c432084ef4414a59b528d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6fa6593e89c432084ef4414a59b528d" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 37147 } } peers { permanent_uuid: "4cfba59d25694d2e8403bb8980334d9b" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 36959 } } peers { permanent_uuid: "42f4ffc7e1c743ecae94261eca355f2f" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 35239 } }
I20260928 21:58:07.142181 22929 ts_tablet_manager.cc:1428] T bc11e362a8474b87ae137bcc0bb563de P e6fa6593e89c432084ef4414a59b528d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 21:58:07.142212 22932 tablet_bootstrap.cc:492] T bc11e362a8474b87ae137bcc0bb563de P 4cfba59d25694d2e8403bb8980334d9b: Bootstrap starting.
I20260928 21:58:07.142294 22914 heartbeater.cc:503] Master 127.2.182.126:36291 was elected leader, sending a full tablet report...
I20260928 21:58:07.142375 22928 raft_consensus.cc:348] T bc11e362a8474b87ae137bcc0bb563de P 42f4ffc7e1c743ecae94261eca355f2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6fa6593e89c432084ef4414a59b528d" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 37147 } } peers { permanent_uuid: "4cfba59d25694d2e8403bb8980334d9b" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 36959 } } peers { permanent_uuid: "42f4ffc7e1c743ecae94261eca355f2f" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 35239 } }
I20260928 21:58:07.142532 22928 raft_consensus.cc:374] T bc11e362a8474b87ae137bcc0bb563de P 42f4ffc7e1c743ecae94261eca355f2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:58:07.142572 22928 raft_consensus.cc:729] T bc11e362a8474b87ae137bcc0bb563de P 42f4ffc7e1c743ecae94261eca355f2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42f4ffc7e1c743ecae94261eca355f2f, State: Initialized, Role: FOLLOWER
I20260928 21:58:07.142680 22928 consensus_queue.cc:260] T bc11e362a8474b87ae137bcc0bb563de P 42f4ffc7e1c743ecae94261eca355f2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6fa6593e89c432084ef4414a59b528d" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 37147 } } peers { permanent_uuid: "4cfba59d25694d2e8403bb8980334d9b" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 36959 } } peers { permanent_uuid: "42f4ffc7e1c743ecae94261eca355f2f" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 35239 } }
I20260928 21:58:07.142906 22932 tablet_bootstrap.cc:654] T bc11e362a8474b87ae137bcc0bb563de P 4cfba59d25694d2e8403bb8980334d9b: Neither blocks nor log segments found. Creating new log.
I20260928 21:58:07.142910 22928 ts_tablet_manager.cc:1428] T bc11e362a8474b87ae137bcc0bb563de P 42f4ffc7e1c743ecae94261eca355f2f: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 21:58:07.142959 22783 heartbeater.cc:503] Master 127.2.182.126:36291 was elected leader, sending a full tablet report...
I20260928 21:58:07.143164 22932 log.cc:824] T bc11e362a8474b87ae137bcc0bb563de P 4cfba59d25694d2e8403bb8980334d9b: Log is configured to *not* fsync() on all Append() calls
I20260928 21:58:07.143838 22932 tablet_bootstrap.cc:492] T bc11e362a8474b87ae137bcc0bb563de P 4cfba59d25694d2e8403bb8980334d9b: No bootstrap required, opened a new log
I20260928 21:58:07.143896 22932 ts_tablet_manager.cc:1397] T bc11e362a8474b87ae137bcc0bb563de P 4cfba59d25694d2e8403bb8980334d9b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 21:58:07.145612 22932 raft_consensus.cc:348] T bc11e362a8474b87ae137bcc0bb563de P 4cfba59d25694d2e8403bb8980334d9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6fa6593e89c432084ef4414a59b528d" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 37147 } } peers { permanent_uuid: "4cfba59d25694d2e8403bb8980334d9b" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 36959 } } peers { permanent_uuid: "42f4ffc7e1c743ecae94261eca355f2f" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 35239 } }
I20260928 21:58:07.145726 22932 raft_consensus.cc:374] T bc11e362a8474b87ae137bcc0bb563de P 4cfba59d25694d2e8403bb8980334d9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:58:07.145756 22932 raft_consensus.cc:729] T bc11e362a8474b87ae137bcc0bb563de P 4cfba59d25694d2e8403bb8980334d9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cfba59d25694d2e8403bb8980334d9b, State: Initialized, Role: FOLLOWER
I20260928 21:58:07.145859 22932 consensus_queue.cc:260] T bc11e362a8474b87ae137bcc0bb563de P 4cfba59d25694d2e8403bb8980334d9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6fa6593e89c432084ef4414a59b528d" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 37147 } } peers { permanent_uuid: "4cfba59d25694d2e8403bb8980334d9b" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 36959 } } peers { permanent_uuid: "42f4ffc7e1c743ecae94261eca355f2f" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 35239 } }
I20260928 21:58:07.146064 22932 ts_tablet_manager.cc:1428] T bc11e362a8474b87ae137bcc0bb563de P 4cfba59d25694d2e8403bb8980334d9b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 21:58:07.146124 22652 heartbeater.cc:503] Master 127.2.182.126:36291 was elected leader, sending a full tablet report...
W20260928 21:58:07.226612 22784 tablet.cc:2369] T bc11e362a8474b87ae137bcc0bb563de P 42f4ffc7e1c743ecae94261eca355f2f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 21:58:07.349753 22653 tablet.cc:2369] T bc11e362a8474b87ae137bcc0bb563de P 4cfba59d25694d2e8403bb8980334d9b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 21:58:07.361258 22915 tablet.cc:2369] T bc11e362a8474b87ae137bcc0bb563de P e6fa6593e89c432084ef4414a59b528d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:58:07.368804 22934 raft_consensus.cc:482] T bc11e362a8474b87ae137bcc0bb563de P 42f4ffc7e1c743ecae94261eca355f2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:58:07.368966 22934 raft_consensus.cc:504] T bc11e362a8474b87ae137bcc0bb563de P 42f4ffc7e1c743ecae94261eca355f2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6fa6593e89c432084ef4414a59b528d" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 37147 } } peers { permanent_uuid: "4cfba59d25694d2e8403bb8980334d9b" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 36959 } } peers { permanent_uuid: "42f4ffc7e1c743ecae94261eca355f2f" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 35239 } }
I20260928 21:58:07.369339 22934 leader_election.cc:288] T bc11e362a8474b87ae137bcc0bb563de P 42f4ffc7e1c743ecae94261eca355f2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e6fa6593e89c432084ef4414a59b528d (127.2.182.67:37147), 4cfba59d25694d2e8403bb8980334d9b (127.2.182.65:36959)
I20260928 21:58:07.373528 22785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:07.369546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.65:36959 (local address 127.2.182.66:38087)
0928 21:58:07.369602 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:07.369613 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:07.369656 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:07.369859 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:07.369860 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:07.369861 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:58:07.369935 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:07.369937 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:07.370851 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:07.370852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:07.372331 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:07.372335 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:07.373211 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:07.373212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:07.373298 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:07.373303 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 21:58:07.373367 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 21:58:07.373434 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":60}
I20260928 21:58:07.373551 22926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:07.369691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.66:43553 (local address 127.2.182.67:37147)
0928 21:58:07.369738 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:07.369740 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:07.369854 (+ 114us) server_negotiation.cc:407] Connection header received
0928 21:58:07.369905 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:07.369906 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:07.369913 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:07.369952 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:58:07.370100 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:07.370778 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:07.372491 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:07.373121 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:07.373184 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:07.373193 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:07.373418 (+ 225us) server_negotiation.cc:299] Negotiation successful
0928 21:58:07.373471 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 21:58:07.373612 22924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:07.369686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.182.66:38087 (local address 127.2.182.65:36959)
0928 21:58:07.369742 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:07.369744 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:07.369751 (+ 7us) server_negotiation.cc:407] Connection header received
0928 21:58:07.369779 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:07.369780 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:07.369791 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:07.369832 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:58:07.370009 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:07.370750 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:07.372492 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:07.373094 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:07.373184 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:07.373193 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:07.373418 (+ 225us) server_negotiation.cc:299] Negotiation successful
0928 21:58:07.373469 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 21:58:07.373533 22937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:07.369556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.67:37147 (local address 127.2.182.66:43553)
0928 21:58:07.369770 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:07.369780 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:07.369852 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:07.369971 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:07.369971 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:07.369973 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:58:07.370034 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:07.370035 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:07.370852 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:07.370853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:07.372334 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:07.372335 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:07.373212 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:07.373212 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:07.373298 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:07.373303 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 21:58:07.373367 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 21:58:07.373425 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":66,"thread_start_us":99,"threads_started":1}
I20260928 21:58:07.373939 22606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc11e362a8474b87ae137bcc0bb563de" candidate_uuid: "42f4ffc7e1c743ecae94261eca355f2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cfba59d25694d2e8403bb8980334d9b" is_pre_election: true
I20260928 21:58:07.374110 22606 raft_consensus.cc:2454] T bc11e362a8474b87ae137bcc0bb563de P 4cfba59d25694d2e8403bb8980334d9b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42f4ffc7e1c743ecae94261eca355f2f in term 0.
I20260928 21:58:07.374099 22868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc11e362a8474b87ae137bcc0bb563de" candidate_uuid: "42f4ffc7e1c743ecae94261eca355f2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6fa6593e89c432084ef4414a59b528d" is_pre_election: true
I20260928 21:58:07.374225 22868 raft_consensus.cc:2454] T bc11e362a8474b87ae137bcc0bb563de P e6fa6593e89c432084ef4414a59b528d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42f4ffc7e1c743ecae94261eca355f2f in term 0.
I20260928 21:58:07.374377 22670 leader_election.cc:302] T bc11e362a8474b87ae137bcc0bb563de P 42f4ffc7e1c743ecae94261eca355f2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42f4ffc7e1c743ecae94261eca355f2f, 4cfba59d25694d2e8403bb8980334d9b; no voters:
I20260928 21:58:07.374562 22934 raft_consensus.cc:2781] T bc11e362a8474b87ae137bcc0bb563de P 42f4ffc7e1c743ecae94261eca355f2f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:58:07.374605 22934 raft_consensus.cc:482] T bc11e362a8474b87ae137bcc0bb563de P 42f4ffc7e1c743ecae94261eca355f2f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:58:07.374619 22934 raft_consensus.cc:3037] T bc11e362a8474b87ae137bcc0bb563de P 42f4ffc7e1c743ecae94261eca355f2f [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:58:07.375355 22934 raft_consensus.cc:504] T bc11e362a8474b87ae137bcc0bb563de P 42f4ffc7e1c743ecae94261eca355f2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6fa6593e89c432084ef4414a59b528d" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 37147 } } peers { permanent_uuid: "4cfba59d25694d2e8403bb8980334d9b" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 36959 } } peers { permanent_uuid: "42f4ffc7e1c743ecae94261eca355f2f" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 35239 } }
I20260928 21:58:07.375484 22934 leader_election.cc:288] T bc11e362a8474b87ae137bcc0bb563de P 42f4ffc7e1c743ecae94261eca355f2f [CANDIDATE]: Term 1 election: Requested vote from peers e6fa6593e89c432084ef4414a59b528d (127.2.182.67:37147), 4cfba59d25694d2e8403bb8980334d9b (127.2.182.65:36959)
I20260928 21:58:07.375655 22606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc11e362a8474b87ae137bcc0bb563de" candidate_uuid: "42f4ffc7e1c743ecae94261eca355f2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cfba59d25694d2e8403bb8980334d9b"
I20260928 21:58:07.375756 22868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc11e362a8474b87ae137bcc0bb563de" candidate_uuid: "42f4ffc7e1c743ecae94261eca355f2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6fa6593e89c432084ef4414a59b528d"
I20260928 21:58:07.375752 22606 raft_consensus.cc:3037] T bc11e362a8474b87ae137bcc0bb563de P 4cfba59d25694d2e8403bb8980334d9b [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:58:07.375833 22868 raft_consensus.cc:3037] T bc11e362a8474b87ae137bcc0bb563de P e6fa6593e89c432084ef4414a59b528d [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:58:07.376698 22606 raft_consensus.cc:2454] T bc11e362a8474b87ae137bcc0bb563de P 4cfba59d25694d2e8403bb8980334d9b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42f4ffc7e1c743ecae94261eca355f2f in term 1.
I20260928 21:58:07.376698 22868 raft_consensus.cc:2454] T bc11e362a8474b87ae137bcc0bb563de P e6fa6593e89c432084ef4414a59b528d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42f4ffc7e1c743ecae94261eca355f2f in term 1.
I20260928 21:58:07.376883 22670 leader_election.cc:302] T bc11e362a8474b87ae137bcc0bb563de P 42f4ffc7e1c743ecae94261eca355f2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42f4ffc7e1c743ecae94261eca355f2f, 4cfba59d25694d2e8403bb8980334d9b; no voters:
I20260928 21:58:07.376977 22934 raft_consensus.cc:2781] T bc11e362a8474b87ae137bcc0bb563de P 42f4ffc7e1c743ecae94261eca355f2f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:58:07.377168 22934 raft_consensus.cc:686] T bc11e362a8474b87ae137bcc0bb563de P 42f4ffc7e1c743ecae94261eca355f2f [term 1 LEADER]: Becoming Leader. State: Replica: 42f4ffc7e1c743ecae94261eca355f2f, State: Running, Role: LEADER
I20260928 21:58:07.377271 22934 consensus_queue.cc:237] T bc11e362a8474b87ae137bcc0bb563de P 42f4ffc7e1c743ecae94261eca355f2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6fa6593e89c432084ef4414a59b528d" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 37147 } } peers { permanent_uuid: "4cfba59d25694d2e8403bb8980334d9b" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 36959 } } peers { permanent_uuid: "42f4ffc7e1c743ecae94261eca355f2f" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 35239 } }
I20260928 21:58:07.378187 22464 catalog_manager.cc:5495] T bc11e362a8474b87ae137bcc0bb563de P 42f4ffc7e1c743ecae94261eca355f2f reported cstate change: term changed from 0 to 1, leader changed from <none> to 42f4ffc7e1c743ecae94261eca355f2f (127.2.182.66). New cstate: current_term: 1 leader_uuid: "42f4ffc7e1c743ecae94261eca355f2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6fa6593e89c432084ef4414a59b528d" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 37147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cfba59d25694d2e8403bb8980334d9b" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 36959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42f4ffc7e1c743ecae94261eca355f2f" member_type: VOTER last_known_addr { host: "127.2.182.66" port: 35239 } health_report { overall_health: HEALTHY } } }
I20260928 21:58:07.419315 22927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:07.416897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43448 (local address 127.2.182.66:35239)
0928 21:58:07.416963 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:07.416965 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:07.416972 (+ 7us) server_negotiation.cc:407] Connection header received
0928 21:58:07.417010 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:07.417011 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:07.417027 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:07.417117 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:58:07.417233 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:07.417717 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:07.418491 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:07.418697 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:07.418749 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:07.418968 (+ 219us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:58:07.419074 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:58:07.419145 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:07.419189 (+ 44us) server_negotiation.cc:299] Negotiation successful
0928 21:58:07.419206 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 21:58:07.421211 22606 raft_consensus.cc:1261] T bc11e362a8474b87ae137bcc0bb563de P 4cfba59d25694d2e8403bb8980334d9b [term 1 FOLLOWER]: Refusing update from remote peer 42f4ffc7e1c743ecae94261eca355f2f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:58:07.421211 22868 raft_consensus.cc:1261] T bc11e362a8474b87ae137bcc0bb563de P e6fa6593e89c432084ef4414a59b528d [term 1 FOLLOWER]: Refusing update from remote peer 42f4ffc7e1c743ecae94261eca355f2f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:58:07.421584 22934 consensus_queue.cc:1035] T bc11e362a8474b87ae137bcc0bb563de P 42f4ffc7e1c743ecae94261eca355f2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cfba59d25694d2e8403bb8980334d9b" member_type: VOTER last_known_addr { host: "127.2.182.65" port: 36959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:58:07.421711 22938 consensus_queue.cc:1035] T bc11e362a8474b87ae137bcc0bb563de P 42f4ffc7e1c743ecae94261eca355f2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6fa6593e89c432084ef4414a59b528d" member_type: VOTER last_known_addr { host: "127.2.182.67" port: 37147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:58:07.424821 22942 mvcc.cc:205] Tried to move back new op lower bound from 7334431487674544128 to 7334431487497846784. Current Snapshot: MvccSnapshot[applied={T|T < 7334431487674544128}]
I20260928 21:58:07.424638 22943 mvcc.cc:205] Tried to move back new op lower bound from 7334431487674544128 to 7334431487497846784. Current Snapshot: MvccSnapshot[applied={T|T < 7334431487674544128 or (T in {7334431487674544128})}]
I20260928 21:58:07.426116 22463 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:58:07.426199 22463 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:58:07.426910 22463 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fe81677b40c04463b85b9ff048426c75: Sending DeleteTablet for 3 replicas of tablet bc11e362a8474b87ae137bcc0bb563de
I20260928 21:58:07.427201 22848 tablet_service.cc:1514] Processing DeleteTablet for tablet bc11e362a8474b87ae137bcc0bb563de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:58:07 UTC) from {username='slave'} at 127.0.0.1:57448
I20260928 21:58:07.427237 22717 tablet_service.cc:1514] Processing DeleteTablet for tablet bc11e362a8474b87ae137bcc0bb563de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:58:07 UTC) from {username='slave'} at 127.0.0.1:43432
I20260928 21:58:07.427444 22952 tablet_replica.cc:330] T bc11e362a8474b87ae137bcc0bb563de P 42f4ffc7e1c743ecae94261eca355f2f: stopping tablet replica
I20260928 21:58:07.427484 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 22524
I20260928 21:58:07.427534 22952 raft_consensus.cc:2229] T bc11e362a8474b87ae137bcc0bb563de P 42f4ffc7e1c743ecae94261eca355f2f [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:58:07.428488 22952 raft_consensus.cc:2258] T bc11e362a8474b87ae137bcc0bb563de P 42f4ffc7e1c743ecae94261eca355f2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:58:07.428735 22951 tablet_replica.cc:330] T bc11e362a8474b87ae137bcc0bb563de P e6fa6593e89c432084ef4414a59b528d: stopping tablet replica
I20260928 21:58:07.428840 22951 raft_consensus.cc:2229] T bc11e362a8474b87ae137bcc0bb563de P e6fa6593e89c432084ef4414a59b528d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:58:07.428947 22951 raft_consensus.cc:2258] T bc11e362a8474b87ae137bcc0bb563de P e6fa6593e89c432084ef4414a59b528d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:58:07.429147 22952 ts_tablet_manager.cc:1905] T bc11e362a8474b87ae137bcc0bb563de P 42f4ffc7e1c743ecae94261eca355f2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:58:07.429389 22951 ts_tablet_manager.cc:1905] T bc11e362a8474b87ae137bcc0bb563de P e6fa6593e89c432084ef4414a59b528d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:58:07.431028 22952 ts_tablet_manager.cc:1918] T bc11e362a8474b87ae137bcc0bb563de P 42f4ffc7e1c743ecae94261eca355f2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:58:07.431103 22952 log.cc:1196] T bc11e362a8474b87ae137bcc0bb563de P 42f4ffc7e1c743ecae94261eca355f2f: Deleting WAL directory at /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632440055392-2777-0/minicluster-data/ts-1/wal/wals/bc11e362a8474b87ae137bcc0bb563de
I20260928 21:58:07.431269 22951 ts_tablet_manager.cc:1918] T bc11e362a8474b87ae137bcc0bb563de P e6fa6593e89c432084ef4414a59b528d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:58:07.431320 22952 ts_tablet_manager.cc:1939] T bc11e362a8474b87ae137bcc0bb563de P 42f4ffc7e1c743ecae94261eca355f2f: Deleting consensus metadata
I20260928 21:58:07.431449 22951 log.cc:1196] T bc11e362a8474b87ae137bcc0bb563de P e6fa6593e89c432084ef4414a59b528d: Deleting WAL directory at /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632440055392-2777-0/minicluster-data/ts-2/wal/wals/bc11e362a8474b87ae137bcc0bb563de
I20260928 21:58:07.431739 22951 ts_tablet_manager.cc:1939] T bc11e362a8474b87ae137bcc0bb563de P e6fa6593e89c432084ef4414a59b528d: Deleting consensus metadata
I20260928 21:58:07.431717 22448 catalog_manager.cc:4841] TS 42f4ffc7e1c743ecae94261eca355f2f (127.2.182.66:35239): tablet bc11e362a8474b87ae137bcc0bb563de (table test-table [id=a57bf8024e3d4410a41ee2a4ade5f4e1]) successfully deleted
I20260928 21:58:07.432173 22449 catalog_manager.cc:4841] TS e6fa6593e89c432084ef4414a59b528d (127.2.182.67:37147): tablet bc11e362a8474b87ae137bcc0bb563de (table test-table [id=a57bf8024e3d4410a41ee2a4ade5f4e1]) successfully deleted
W20260928 21:58:07.433426 22448 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.182.65:36959 (error 108)
I20260928 21:58:07.433660 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 22655
I20260928 21:58:07.434176 22922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:07.433770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.65:36959 (local address 127.0.0.1:59298)
0928 21:58:07.434100 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:07.434148 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.65:36959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310}
W20260928 21:58:07.434403 22448 catalog_manager.cc:4584] TS 4cfba59d25694d2e8403bb8980334d9b (127.2.182.65:36959): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bc11e362a8474b87ae137bcc0bb563de: Network error: Client connection negotiation failed: client connection to 127.2.182.65:36959: connect: Connection refused (error 111)
I20260928 21:58:07.438406 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 22786
I20260928 21:58:07.442452 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 22433
2026-09-28T21:58:07Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (889 ms)
[----------] 6 tests from EncryptionPolicyTest (8103 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632440055392-2777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:58:07 dist-test-slave-k6lz krb5kdc[22956](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:58:07 dist-test-slave-k6lz krb5kdc[22956](info): set up 2 sockets
Sep 28 21:58:07 dist-test-slave-k6lz krb5kdc[22956](info): commencing operation
krb5kdc: starting...
W20260928 21:58:08.411127 2777 mini_kdc.cc:121] Time spent starting KDC: real 0.950s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Sep 28 21:58:08 dist-test-slave-k6lz krb5kdc[22956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:58:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:58:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632440055392-2777-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632440055392-2777-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 21:58:08.527736 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632440055392-2777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632440055392-2777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632440055392-2777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42905
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632440055392-2777-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42905
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:58:08.607717 22969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:58:08.607924 22969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:58:08.607954 22969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:58:08.609187 22969 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:58:08.609228 22969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:58:08.609241 22969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:58:08.609252 22969 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:58:08.609262 22969 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:58:08.610677 22969 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632440055392-2777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632440055392-2777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42905
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632440055392-2777-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632440055392-2777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:58:08.610870 22969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:58:08.611097 22969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:58:08.616165 22975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:08.616230 22977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:08.616289 22974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:08.616289 22969 server_base.cc:1034] running on GCE node
I20260928 21:58:08.616663 22969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:58:08.616930 22969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:58:08.618090 22969 hybrid_clock.cc:648] HybridClock initialized: now 1790632688618069 us; error 39 us; skew 500 ppm
Sep 28 21:58:08 dist-test-slave-k6lz krb5kdc[22956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632688, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:58:08.620680 22969 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 21:58:08.621173 22969 webserver.cc:460] Webserver started at http://127.0.0.1:46589/ using document root <none> and password file <none>
I20260928 21:58:08.621373 22969 fs_manager.cc:359] Metadata directory not provided
I20260928 21:58:08.621418 22969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:58:08.621539 22969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:58:08.622602 22969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632440055392-2777-0/minicluster-data/master-0/data/instance:
uuid: "e5e960b24dd7489c97943d577052829e"
format_stamp: "Formatted at 2026-09-28 21:58:08 on dist-test-slave-k6lz"
server_key: "1c4b7330dcc90c28277429ad68bf17b6"
server_key_iv: "a910a8aabe9891ed4ae5b53f09839ebc"
server_key_version: "encryptionkey@0"
I20260928 21:58:08.622920 22969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632440055392-2777-0/minicluster-data/master-0/wal/instance:
uuid: "e5e960b24dd7489c97943d577052829e"
format_stamp: "Formatted at 2026-09-28 21:58:08 on dist-test-slave-k6lz"
server_key: "1c4b7330dcc90c28277429ad68bf17b6"
server_key_iv: "a910a8aabe9891ed4ae5b53f09839ebc"
server_key_version: "encryptionkey@0"
I20260928 21:58:08.624176 22969 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 21:58:08.624960 22984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:08.625128 22969 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 21:58:08.625207 22969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632440055392-2777-0/minicluster-data/master-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632440055392-2777-0/minicluster-data/master-0/wal
uuid: "e5e960b24dd7489c97943d577052829e"
format_stamp: "Formatted at 2026-09-28 21:58:08 on dist-test-slave-k6lz"
server_key: "1c4b7330dcc90c28277429ad68bf17b6"
server_key_iv: "a910a8aabe9891ed4ae5b53f09839ebc"
server_key_version: "encryptionkey@0"
I20260928 21:58:08.625264 22969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632440055392-2777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632440055392-2777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632440055392-2777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:58:08.637346 22969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:58:08.639482 22969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:58:08.639632 22969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:58:08.643491 22969 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42905
I20260928 21:58:08.643539 23036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42905 every 8 connection(s)
I20260928 21:58:08.643860 22969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
I20260928 21:58:08.644790 23037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:58:08.647194 23037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5e960b24dd7489c97943d577052829e: Bootstrap starting.
I20260928 21:58:08.647775 23037 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5e960b24dd7489c97943d577052829e: Neither blocks nor log segments found. Creating new log.
I20260928 21:58:08.648046 23037 log.cc:824] T 00000000000000000000000000000000 P e5e960b24dd7489c97943d577052829e: Log is configured to *not* fsync() on all Append() calls
I20260928 21:58:08.648737 23037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5e960b24dd7489c97943d577052829e: No bootstrap required, opened a new log
I20260928 21:58:08.650377 23037 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5e960b24dd7489c97943d577052829e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5e960b24dd7489c97943d577052829e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42905 } }
I20260928 21:58:08.650519 23037 raft_consensus.cc:374] T 00000000000000000000000000000000 P e5e960b24dd7489c97943d577052829e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:58:08.650542 23037 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5e960b24dd7489c97943d577052829e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5e960b24dd7489c97943d577052829e, State: Initialized, Role: FOLLOWER
I20260928 21:58:08.650665 23037 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5e960b24dd7489c97943d577052829e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5e960b24dd7489c97943d577052829e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42905 } }
I20260928 21:58:08.650732 23037 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5e960b24dd7489c97943d577052829e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:58:08.650754 23037 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5e960b24dd7489c97943d577052829e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:58:08.650785 23037 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5e960b24dd7489c97943d577052829e [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:58:08.651300 23037 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5e960b24dd7489c97943d577052829e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5e960b24dd7489c97943d577052829e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42905 } }
I20260928 21:58:08.651428 23037 leader_election.cc:302] T 00000000000000000000000000000000 P e5e960b24dd7489c97943d577052829e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5e960b24dd7489c97943d577052829e; no voters:
I20260928 21:58:08.651587 23037 leader_election.cc:288] T 00000000000000000000000000000000 P e5e960b24dd7489c97943d577052829e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:58:08.651626 23040 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5e960b24dd7489c97943d577052829e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:58:08.651760 23040 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5e960b24dd7489c97943d577052829e [term 1 LEADER]: Becoming Leader. State: Replica: e5e960b24dd7489c97943d577052829e, State: Running, Role: LEADER
I20260928 21:58:08.651829 23037 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5e960b24dd7489c97943d577052829e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:58:08.651845 23040 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5e960b24dd7489c97943d577052829e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5e960b24dd7489c97943d577052829e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42905 } }
I20260928 21:58:08.652227 23042 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5e960b24dd7489c97943d577052829e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5e960b24dd7489c97943d577052829e. Latest consensus state: current_term: 1 leader_uuid: "e5e960b24dd7489c97943d577052829e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5e960b24dd7489c97943d577052829e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42905 } } }
I20260928 21:58:08.652321 23042 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5e960b24dd7489c97943d577052829e [sys.catalog]: This master's current role is: LEADER
I20260928 21:58:08.652455 23041 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5e960b24dd7489c97943d577052829e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e5e960b24dd7489c97943d577052829e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5e960b24dd7489c97943d577052829e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42905 } } }
I20260928 21:58:08.652633 23041 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5e960b24dd7489c97943d577052829e [sys.catalog]: This master's current role is: LEADER
I20260928 21:58:08.652628 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 22969
I20260928 21:58:08.652724 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632440055392-2777-0/minicluster-data/master-0/wal/instance
I20260928 21:58:08.652908 2777 external_mini_cluster.cc:1428] Setting key 3661591af6e326020d5e038742953d9c
W20260928 21:58:08.653568 23056 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e5e960b24dd7489c97943d577052829e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:58:08.653626 23056 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:58:08.653728 23049 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:58:08.653923 23049 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:58:08.655434 23049 catalog_manager.cc:1347] Generated new cluster ID: 3c4620ab16c643e3b2ae06307c3ea6ea
I20260928 21:58:08.655488 23049 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 21:58:08 dist-test-slave-k6lz krb5kdc[22956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 21:58:08.662377 23049 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:58:08.662835 23059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:08.653463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48143 (local address 127.0.0.1:42905)
0928 21:58:08.653612 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:08.653621 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:08.653655 (+ 34us) server_negotiation.cc:407] Connection header received
0928 21:58:08.653803 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:08.653810 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:08.653867 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:08.654004 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:08.654654 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:08.655750 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:08.656455 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:08.656861 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:08.656954 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:08.659096 (+ 2142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:08.659100 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:08.659100 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:08.659112 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:08.661164 (+ 2052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:08.661568 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:08.661571 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:08.661572 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:08.661675 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:08.661876 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:08.661877 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:08.661877 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:08.662158 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:08.662319 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:08.662505 (+ 186us) server_negotiation.cc:299] Negotiation successful
0928 21:58:08.662619 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":9,"server-negotiator.queue_time_us":107,"thread_start_us":65,"threads_started":1}
I20260928 21:58:08.663034 23049 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:58:08.663700 2777 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:58:08.671013 23049 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e5e960b24dd7489c97943d577052829e: Generated new TSK 0
I20260928 21:58:08.671205 23049 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:58:08.672075 23059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:08.665273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36017 (local address 127.0.0.1:42905)
0928 21:58:08.665305 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:08.665306 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:08.665315 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:58:08.665359 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:08.665360 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:08.665368 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:08.665424 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:08.666016 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:08.666790 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:08.667840 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:08.668155 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:08.668210 (+ 55us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:08.669515 (+ 1305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:08.669516 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:08.669516 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:08.669522 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:08.670934 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:08.671352 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:08.671353 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:08.671354 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:08.671412 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:08.671605 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:08.671606 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:08.671606 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:08.671717 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:08.671803 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:08.671932 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 21:58:08.671992 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 21:58:08.673702 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 22969
W20260928 21:58:08.677696 22954 connection.cc:535] client connection to 127.0.0.1:42905 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42905): Connection reset by peer (error 104)
2026-09-28T21:58:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1230 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632440055392-2777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:58:08 dist-test-slave-k6lz krb5kdc[23068](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:58:08 dist-test-slave-k6lz krb5kdc[23068](info): set up 2 sockets
Sep 28 21:58:08 dist-test-slave-k6lz krb5kdc[23068](info): commencing operation
krb5kdc: starting...
W20260928 21:58:09.635195 2777 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Sep 28 21:58:09 dist-test-slave-k6lz krb5kdc[23068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632689, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:58:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:58:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632440055392-2777-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632440055392-2777-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 21:58:09.750705 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632440055392-2777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632440055392-2777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632440055392-2777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39659
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632440055392-2777-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39659
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:58:09.831024 23081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:58:09.831251 23081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:58:09.831286 23081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:58:09.832551 23081 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:58:09.832605 23081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:58:09.832623 23081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:58:09.832641 23081 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:58:09.832656 23081 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:58:09.834182 23081 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632440055392-2777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632440055392-2777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39659
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632440055392-2777-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632440055392-2777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:58:09.834461 23081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:58:09.834676 23081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:58:09.839594 23086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:09.839773 23081 server_base.cc:1034] running on GCE node
W20260928 21:58:09.839589 23087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:09.839600 23089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:09.840121 23081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:58:09.840412 23081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:58:09.841578 23081 hybrid_clock.cc:648] HybridClock initialized: now 1790632689841564 us; error 31 us; skew 500 ppm
Sep 28 21:58:09 dist-test-slave-k6lz krb5kdc[23068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632689, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:58:09.848899 23081 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 21:58:09.849406 23081 webserver.cc:460] Webserver started at http://127.0.0.1:41175/ using document root <none> and password file <none>
I20260928 21:58:09.849620 23081 fs_manager.cc:359] Metadata directory not provided
I20260928 21:58:09.849670 23081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:58:09.849797 23081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:58:09.850862 23081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632440055392-2777-0/minicluster-data/master-0/data/instance:
uuid: "6984e2732a59449ebe1f46d0f47cd7e0"
format_stamp: "Formatted at 2026-09-28 21:58:09 on dist-test-slave-k6lz"
server_key: "999ae400da20240fe7306755bbc5e46c"
server_key_iv: "90e9d2481a04b54199e2285275709ee3"
server_key_version: "encryptionkey@0"
I20260928 21:58:09.851192 23081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632440055392-2777-0/minicluster-data/master-0/wal/instance:
uuid: "6984e2732a59449ebe1f46d0f47cd7e0"
format_stamp: "Formatted at 2026-09-28 21:58:09 on dist-test-slave-k6lz"
server_key: "999ae400da20240fe7306755bbc5e46c"
server_key_iv: "90e9d2481a04b54199e2285275709ee3"
server_key_version: "encryptionkey@0"
I20260928 21:58:09.852596 23081 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:58:09.853459 23096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:09.853699 23081 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:58:09.853798 23081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632440055392-2777-0/minicluster-data/master-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632440055392-2777-0/minicluster-data/master-0/wal
uuid: "6984e2732a59449ebe1f46d0f47cd7e0"
format_stamp: "Formatted at 2026-09-28 21:58:09 on dist-test-slave-k6lz"
server_key: "999ae400da20240fe7306755bbc5e46c"
server_key_iv: "90e9d2481a04b54199e2285275709ee3"
server_key_version: "encryptionkey@0"
I20260928 21:58:09.853873 23081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632440055392-2777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632440055392-2777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632440055392-2777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:58:09.872231 23081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:58:09.874639 23081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:58:09.874789 23081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:58:09.878850 23081 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39659
I20260928 21:58:09.878970 23148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39659 every 8 connection(s)
I20260928 21:58:09.879241 23081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
I20260928 21:58:09.880707 23149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:58:09.882974 23149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6984e2732a59449ebe1f46d0f47cd7e0: Bootstrap starting.
I20260928 21:58:09.883553 23149 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6984e2732a59449ebe1f46d0f47cd7e0: Neither blocks nor log segments found. Creating new log.
I20260928 21:58:09.883780 23149 log.cc:824] T 00000000000000000000000000000000 P 6984e2732a59449ebe1f46d0f47cd7e0: Log is configured to *not* fsync() on all Append() calls
I20260928 21:58:09.884459 23149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6984e2732a59449ebe1f46d0f47cd7e0: No bootstrap required, opened a new log
I20260928 21:58:09.885542 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 23081
I20260928 21:58:09.885653 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632440055392-2777-0/minicluster-data/master-0/wal/instance
I20260928 21:58:09.885830 2777 external_mini_cluster.cc:1428] Setting key b3b0ce2af00a0e25cd1a4d7f91efce46
I20260928 21:58:09.886400 23149 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6984e2732a59449ebe1f46d0f47cd7e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6984e2732a59449ebe1f46d0f47cd7e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39659 } }
I20260928 21:58:09.886579 23149 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6984e2732a59449ebe1f46d0f47cd7e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:58:09.886607 23149 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6984e2732a59449ebe1f46d0f47cd7e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6984e2732a59449ebe1f46d0f47cd7e0, State: Initialized, Role: FOLLOWER
I20260928 21:58:09.886731 23149 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6984e2732a59449ebe1f46d0f47cd7e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6984e2732a59449ebe1f46d0f47cd7e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39659 } }
I20260928 21:58:09.886790 23149 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6984e2732a59449ebe1f46d0f47cd7e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:58:09.886811 23149 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6984e2732a59449ebe1f46d0f47cd7e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:58:09.886857 23149 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6984e2732a59449ebe1f46d0f47cd7e0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:58:09.887822 23149 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6984e2732a59449ebe1f46d0f47cd7e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6984e2732a59449ebe1f46d0f47cd7e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39659 } }
I20260928 21:58:09.887912 23149 leader_election.cc:302] T 00000000000000000000000000000000 P 6984e2732a59449ebe1f46d0f47cd7e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6984e2732a59449ebe1f46d0f47cd7e0; no voters:
I20260928 21:58:09.888067 23149 leader_election.cc:288] T 00000000000000000000000000000000 P 6984e2732a59449ebe1f46d0f47cd7e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:58:09.888114 23154 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6984e2732a59449ebe1f46d0f47cd7e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:58:09.888311 23149 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6984e2732a59449ebe1f46d0f47cd7e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:58:09.888310 23154 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6984e2732a59449ebe1f46d0f47cd7e0 [term 1 LEADER]: Becoming Leader. State: Replica: 6984e2732a59449ebe1f46d0f47cd7e0, State: Running, Role: LEADER
I20260928 21:58:09.888532 23154 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6984e2732a59449ebe1f46d0f47cd7e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6984e2732a59449ebe1f46d0f47cd7e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39659 } }
I20260928 21:58:09.888919 23156 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6984e2732a59449ebe1f46d0f47cd7e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6984e2732a59449ebe1f46d0f47cd7e0. Latest consensus state: current_term: 1 leader_uuid: "6984e2732a59449ebe1f46d0f47cd7e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6984e2732a59449ebe1f46d0f47cd7e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39659 } } }
I20260928 21:58:09.888928 23155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6984e2732a59449ebe1f46d0f47cd7e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6984e2732a59449ebe1f46d0f47cd7e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6984e2732a59449ebe1f46d0f47cd7e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39659 } } }
I20260928 21:58:09.889065 23156 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6984e2732a59449ebe1f46d0f47cd7e0 [sys.catalog]: This master's current role is: LEADER
I20260928 21:58:09.889088 23155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6984e2732a59449ebe1f46d0f47cd7e0 [sys.catalog]: This master's current role is: LEADER
I20260928 21:58:09.889249 23163 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:58:09.889765 23163 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:58:09.891402 23153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:09.886377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35869 (local address 127.0.0.1:39659)
0928 21:58:09.886546 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:09.886561 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:09.886592 (+ 31us) server_negotiation.cc:407] Connection header received
0928 21:58:09.886707 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:09.886715 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:09.886773 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:09.886913 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:09.887116 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:09.888220 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:09.889260 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:09.889854 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:09.890274 (+ 420us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:09.890673 (+ 399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:09.890677 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:09.890677 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:58:09.890869 (+ 192us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:09.890905 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:58:09.890919 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:09.891001 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:09.891143 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 21:58:09.891169 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":81,"threads_started":1}
I20260928 21:58:09.892170 23163 catalog_manager.cc:1347] Generated new cluster ID: 05c8e052290e40a3be26df196e9ae442
I20260928 21:58:09.892218 23163 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:58:09.892400 2777 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:58:09.898993 23153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:09.893802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42225 (local address 127.0.0.1:39659)
0928 21:58:09.893832 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:09.893833 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:09.895619 (+ 1786us) server_negotiation.cc:407] Connection header received
0928 21:58:09.895659 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:09.895661 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:09.895674 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:09.895745 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:09.895888 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:09.896632 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:09.897620 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:09.897949 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:09.898020 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:09.898740 (+ 720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:09.898742 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:09.898742 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:58:09.898747 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:09.898766 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:58:09.898771 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:09.898831 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:09.898890 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 21:58:09.898905 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260928 21:58:09.909034 23163 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:58:09.909574 23163 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:58:09.914054 23163 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6984e2732a59449ebe1f46d0f47cd7e0: Generated new TSK 0
I20260928 21:58:09.914249 23163 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:58:09.918658 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 23081
W20260928 21:58:09.922768 23066 connection.cc:535] client connection to 127.0.0.1:39659 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39659): Connection reset by peer (error 104)
2026-09-28T21:58:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1245 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632440055392-2777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:58:09 dist-test-slave-k6lz krb5kdc[23180](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:58:09 dist-test-slave-k6lz krb5kdc[23180](info): set up 2 sockets
Sep 28 21:58:09 dist-test-slave-k6lz krb5kdc[23180](info): commencing operation
krb5kdc: starting...
W20260928 21:58:10.887192 2777 mini_kdc.cc:121] Time spent starting KDC: real 0.951s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Sep 28 21:58:10 dist-test-slave-k6lz krb5kdc[23180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632690, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:58:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:58:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632440055392-2777-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632440055392-2777-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 21:58:11.011686 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632440055392-2777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632440055392-2777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632440055392-2777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45795
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632440055392-2777-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45795
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:58:11.091825 23193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:58:11.092046 23193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:58:11.092070 23193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:58:11.093214 23193 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:58:11.093253 23193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:58:11.093267 23193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:58:11.093278 23193 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:58:11.093288 23193 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:58:11.094777 23193 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632440055392-2777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632440055392-2777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45795
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632440055392-2777-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632440055392-2777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:58:11.094997 23193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:58:11.095197 23193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:58:11.100155 23193 server_base.cc:1034] running on GCE node
W20260928 21:58:11.100126 23199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:11.100119 23198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:11.100241 23201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:11.100594 23193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:58:11.100910 23193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:58:11.102082 23193 hybrid_clock.cc:648] HybridClock initialized: now 1790632691102062 us; error 33 us; skew 500 ppm
Sep 28 21:58:11 dist-test-slave-k6lz krb5kdc[23180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632691, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:58:11.104738 23193 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 21:58:11.105274 23193 webserver.cc:460] Webserver started at http://127.0.0.1:37433/ using document root <none> and password file <none>
I20260928 21:58:11.105500 23193 fs_manager.cc:359] Metadata directory not provided
I20260928 21:58:11.105548 23193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:58:11.105670 23193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:58:11.106724 23193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632440055392-2777-0/minicluster-data/master-0/data/instance:
uuid: "b6a2620feda74613b4a4c32075beb609"
format_stamp: "Formatted at 2026-09-28 21:58:11 on dist-test-slave-k6lz"
server_key: "7e2511b3c13e2e73c8bd8fe43985c059"
server_key_iv: "0f559d34df7e1a6c619542e75b445433"
server_key_version: "encryptionkey@0"
I20260928 21:58:11.107048 23193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632440055392-2777-0/minicluster-data/master-0/wal/instance:
uuid: "b6a2620feda74613b4a4c32075beb609"
format_stamp: "Formatted at 2026-09-28 21:58:11 on dist-test-slave-k6lz"
server_key: "7e2511b3c13e2e73c8bd8fe43985c059"
server_key_iv: "0f559d34df7e1a6c619542e75b445433"
server_key_version: "encryptionkey@0"
I20260928 21:58:11.108389 23193 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:58:11.109153 23208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:11.109320 23193 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:58:11.109405 23193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632440055392-2777-0/minicluster-data/master-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632440055392-2777-0/minicluster-data/master-0/wal
uuid: "b6a2620feda74613b4a4c32075beb609"
format_stamp: "Formatted at 2026-09-28 21:58:11 on dist-test-slave-k6lz"
server_key: "7e2511b3c13e2e73c8bd8fe43985c059"
server_key_iv: "0f559d34df7e1a6c619542e75b445433"
server_key_version: "encryptionkey@0"
I20260928 21:58:11.109539 23193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632440055392-2777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632440055392-2777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632440055392-2777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:58:11.112900 23193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:58:11.115075 23193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:58:11.115206 23193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:58:11.119117 23193 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45795
I20260928 21:58:11.119156 23260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45795 every 8 connection(s)
I20260928 21:58:11.119519 23193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
I20260928 21:58:11.120534 23261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:58:11.123099 23261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6a2620feda74613b4a4c32075beb609: Bootstrap starting.
I20260928 21:58:11.123687 23261 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6a2620feda74613b4a4c32075beb609: Neither blocks nor log segments found. Creating new log.
I20260928 21:58:11.123922 23261 log.cc:824] T 00000000000000000000000000000000 P b6a2620feda74613b4a4c32075beb609: Log is configured to *not* fsync() on all Append() calls
I20260928 21:58:11.124771 23261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6a2620feda74613b4a4c32075beb609: No bootstrap required, opened a new log
I20260928 21:58:11.126389 23261 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6a2620feda74613b4a4c32075beb609 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6a2620feda74613b4a4c32075beb609" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45795 } }
I20260928 21:58:11.126542 23261 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6a2620feda74613b4a4c32075beb609 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:58:11.126564 23261 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6a2620feda74613b4a4c32075beb609 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6a2620feda74613b4a4c32075beb609, State: Initialized, Role: FOLLOWER
I20260928 21:58:11.126641 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 23193
I20260928 21:58:11.126689 23261 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6a2620feda74613b4a4c32075beb609 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6a2620feda74613b4a4c32075beb609" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45795 } }
I20260928 21:58:11.126751 23261 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6a2620feda74613b4a4c32075beb609 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:58:11.126747 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632440055392-2777-0/minicluster-data/master-0/wal/instance
I20260928 21:58:11.126791 23261 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6a2620feda74613b4a4c32075beb609 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:58:11.126834 23261 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6a2620feda74613b4a4c32075beb609 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:58:11.126938 2777 external_mini_cluster.cc:1428] Setting key 540f3b99eb140459e297a5ce13afea73
I20260928 21:58:11.127581 23261 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6a2620feda74613b4a4c32075beb609 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6a2620feda74613b4a4c32075beb609" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45795 } }
I20260928 21:58:11.127692 23261 leader_election.cc:302] T 00000000000000000000000000000000 P b6a2620feda74613b4a4c32075beb609 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6a2620feda74613b4a4c32075beb609; no voters:
I20260928 21:58:11.127861 23261 leader_election.cc:288] T 00000000000000000000000000000000 P b6a2620feda74613b4a4c32075beb609 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:58:11.127905 23266 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6a2620feda74613b4a4c32075beb609 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:58:11.128106 23266 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6a2620feda74613b4a4c32075beb609 [term 1 LEADER]: Becoming Leader. State: Replica: b6a2620feda74613b4a4c32075beb609, State: Running, Role: LEADER
I20260928 21:58:11.128141 23261 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6a2620feda74613b4a4c32075beb609 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:58:11.128590 23266 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6a2620feda74613b4a4c32075beb609 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6a2620feda74613b4a4c32075beb609" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45795 } }
I20260928 21:58:11.128983 23268 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6a2620feda74613b4a4c32075beb609 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6a2620feda74613b4a4c32075beb609. Latest consensus state: current_term: 1 leader_uuid: "b6a2620feda74613b4a4c32075beb609" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6a2620feda74613b4a4c32075beb609" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45795 } } }
I20260928 21:58:11.129063 23268 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6a2620feda74613b4a4c32075beb609 [sys.catalog]: This master's current role is: LEADER
I20260928 21:58:11.129202 23265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:11.127455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60851 (local address 127.0.0.1:45795)
0928 21:58:11.127629 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:11.127638 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:11.127670 (+ 32us) server_negotiation.cc:407] Connection header received
0928 21:58:11.127806 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:11.127823 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:11.127849 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:11.128258 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:11.128667 (+ 409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:11.128671 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:11.128671 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:58:11.128675 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:11.128719 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:58:11.128735 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:11.128785 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:11.128846 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 21:58:11.128985 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":85,"threads_started":1}
I20260928 21:58:11.129357 23267 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6a2620feda74613b4a4c32075beb609 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6a2620feda74613b4a4c32075beb609" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6a2620feda74613b4a4c32075beb609" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45795 } } }
I20260928 21:58:11.129444 23267 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6a2620feda74613b4a4c32075beb609 [sys.catalog]: This master's current role is: LEADER
I20260928 21:58:11.130025 2777 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260928 21:58:11.130295 23282 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b6a2620feda74613b4a4c32075beb609: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:58:11.130340 23282 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:58:11.130422 23275 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:58:11.130568 23275 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:58:11.132258 23275 catalog_manager.cc:1347] Generated new cluster ID: 569475cb41ca458aac95cdb844ab2256
I20260928 21:58:11.132340 23275 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:58:11.132571 23265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:11.131512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38005 (local address 127.0.0.1:45795)
0928 21:58:11.131546 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:11.131548 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:11.131697 (+ 149us) server_negotiation.cc:407] Connection header received
0928 21:58:11.131752 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:11.131753 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:11.131759 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:11.131809 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:11.132345 (+ 536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:11.132346 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:11.132346 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:58:11.132351 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:11.132367 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:58:11.132371 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:11.132434 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:11.132496 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 21:58:11.132510 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 21:58:11.148211 23275 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:58:11.148389 23275 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:58:11.164553 23275 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6a2620feda74613b4a4c32075beb609: Generated new TSK 0
I20260928 21:58:11.164778 23275 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:58:11.168378 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 23193
2026-09-28T21:58:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1249 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632440055392-2777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:58:11 dist-test-slave-k6lz krb5kdc[23292](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:58:11 dist-test-slave-k6lz krb5kdc[23292](info): set up 2 sockets
Sep 28 21:58:11 dist-test-slave-k6lz krb5kdc[23292](info): commencing operation
krb5kdc: starting...
W20260928 21:58:12.135197 2777 mini_kdc.cc:121] Time spent starting KDC: real 0.949s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Sep 28 21:58:12 dist-test-slave-k6lz krb5kdc[23292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632692, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:58:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:58:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632440055392-2777-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632440055392-2777-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 21:58:12.292266 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632440055392-2777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632440055392-2777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632440055392-2777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43067
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632440055392-2777-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43067
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:58:12.372079 23305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:58:12.372293 23305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:58:12.372325 23305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:58:12.373492 23305 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:58:12.373536 23305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:58:12.373556 23305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:58:12.373574 23305 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:58:12.373589 23305 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:58:12.375102 23305 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632440055392-2777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632440055392-2777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43067
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632440055392-2777-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632440055392-2777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:58:12.375380 23305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:58:12.375602 23305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:58:12.380712 23310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:12.380712 23311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:12.380718 23313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:12.380740 23305 server_base.cc:1034] running on GCE node
I20260928 21:58:12.381179 23305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:58:12.381464 23305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:58:12.382629 23305 hybrid_clock.cc:648] HybridClock initialized: now 1790632692382609 us; error 35 us; skew 500 ppm
Sep 28 21:58:12 dist-test-slave-k6lz krb5kdc[23292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632692, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:58:12.385167 23305 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 21:58:12.385639 23305 webserver.cc:460] Webserver started at http://127.0.0.1:40809/ using document root <none> and password file <none>
I20260928 21:58:12.385841 23305 fs_manager.cc:359] Metadata directory not provided
I20260928 21:58:12.385888 23305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:58:12.386010 23305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:58:12.387152 23305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632440055392-2777-0/minicluster-data/master-0/data/instance:
uuid: "29f23f8e09cf4a3f8c7b0b0fec7085e6"
format_stamp: "Formatted at 2026-09-28 21:58:12 on dist-test-slave-k6lz"
server_key: "2e92e21971304081e31d1f3094fd2e4c"
server_key_iv: "5171f7df6e096cd07677715f1ac1e41f"
server_key_version: "encryptionkey@0"
I20260928 21:58:12.387497 23305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632440055392-2777-0/minicluster-data/master-0/wal/instance:
uuid: "29f23f8e09cf4a3f8c7b0b0fec7085e6"
format_stamp: "Formatted at 2026-09-28 21:58:12 on dist-test-slave-k6lz"
server_key: "2e92e21971304081e31d1f3094fd2e4c"
server_key_iv: "5171f7df6e096cd07677715f1ac1e41f"
server_key_version: "encryptionkey@0"
I20260928 21:58:12.388770 23305 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:58:12.389505 23320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:12.389679 23305 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:58:12.389760 23305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632440055392-2777-0/minicluster-data/master-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632440055392-2777-0/minicluster-data/master-0/wal
uuid: "29f23f8e09cf4a3f8c7b0b0fec7085e6"
format_stamp: "Formatted at 2026-09-28 21:58:12 on dist-test-slave-k6lz"
server_key: "2e92e21971304081e31d1f3094fd2e4c"
server_key_iv: "5171f7df6e096cd07677715f1ac1e41f"
server_key_version: "encryptionkey@0"
I20260928 21:58:12.389827 23305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632440055392-2777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632440055392-2777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632440055392-2777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:58:12.399646 23305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:58:12.401685 23305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:58:12.401818 23305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:58:12.405526 23305 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43067
I20260928 21:58:12.405587 23372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43067 every 8 connection(s)
I20260928 21:58:12.405880 23305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
I20260928 21:58:12.406868 23373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:58:12.407137 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 23305
I20260928 21:58:12.407235 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632440055392-2777-0/minicluster-data/master-0/wal/instance
I20260928 21:58:12.407447 2777 external_mini_cluster.cc:1428] Setting key 04b8c8335b1a6aabc937351abed70466
I20260928 21:58:12.410204 23373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29f23f8e09cf4a3f8c7b0b0fec7085e6: Bootstrap starting.
I20260928 21:58:12.410895 23373 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29f23f8e09cf4a3f8c7b0b0fec7085e6: Neither blocks nor log segments found. Creating new log.
I20260928 21:58:12.411160 23373 log.cc:824] T 00000000000000000000000000000000 P 29f23f8e09cf4a3f8c7b0b0fec7085e6: Log is configured to *not* fsync() on all Append() calls
I20260928 21:58:12.411840 23373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29f23f8e09cf4a3f8c7b0b0fec7085e6: No bootstrap required, opened a new log
I20260928 21:58:12.413843 23373 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29f23f8e09cf4a3f8c7b0b0fec7085e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f23f8e09cf4a3f8c7b0b0fec7085e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43067 } }
I20260928 21:58:12.414042 23373 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29f23f8e09cf4a3f8c7b0b0fec7085e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:58:12.414074 23373 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29f23f8e09cf4a3f8c7b0b0fec7085e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29f23f8e09cf4a3f8c7b0b0fec7085e6, State: Initialized, Role: FOLLOWER
Sep 28 21:58:12 dist-test-slave-k6lz krb5kdc[23292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632692, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 21:58:12.414196 23373 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29f23f8e09cf4a3f8c7b0b0fec7085e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f23f8e09cf4a3f8c7b0b0fec7085e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43067 } }
I20260928 21:58:12.414279 23373 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29f23f8e09cf4a3f8c7b0b0fec7085e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:58:12.414317 23373 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29f23f8e09cf4a3f8c7b0b0fec7085e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:58:12.414382 23373 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29f23f8e09cf4a3f8c7b0b0fec7085e6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:58:12.415035 23373 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29f23f8e09cf4a3f8c7b0b0fec7085e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f23f8e09cf4a3f8c7b0b0fec7085e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43067 } }
I20260928 21:58:12.415139 23373 leader_election.cc:302] T 00000000000000000000000000000000 P 29f23f8e09cf4a3f8c7b0b0fec7085e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29f23f8e09cf4a3f8c7b0b0fec7085e6; no voters:
I20260928 21:58:12.415308 23373 leader_election.cc:288] T 00000000000000000000000000000000 P 29f23f8e09cf4a3f8c7b0b0fec7085e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:58:12.415375 23378 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29f23f8e09cf4a3f8c7b0b0fec7085e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:58:12.415470 23378 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29f23f8e09cf4a3f8c7b0b0fec7085e6 [term 1 LEADER]: Becoming Leader. State: Replica: 29f23f8e09cf4a3f8c7b0b0fec7085e6, State: Running, Role: LEADER
I20260928 21:58:12.415541 23378 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29f23f8e09cf4a3f8c7b0b0fec7085e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f23f8e09cf4a3f8c7b0b0fec7085e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43067 } }
I20260928 21:58:12.415601 23373 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29f23f8e09cf4a3f8c7b0b0fec7085e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:58:12.415840 23378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29f23f8e09cf4a3f8c7b0b0fec7085e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29f23f8e09cf4a3f8c7b0b0fec7085e6. Latest consensus state: current_term: 1 leader_uuid: "29f23f8e09cf4a3f8c7b0b0fec7085e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f23f8e09cf4a3f8c7b0b0fec7085e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43067 } } }
I20260928 21:58:12.415916 23378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29f23f8e09cf4a3f8c7b0b0fec7085e6 [sys.catalog]: This master's current role is: LEADER
I20260928 21:58:12.415941 23379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29f23f8e09cf4a3f8c7b0b0fec7085e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29f23f8e09cf4a3f8c7b0b0fec7085e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f23f8e09cf4a3f8c7b0b0fec7085e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43067 } } }
I20260928 21:58:12.416035 23379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29f23f8e09cf4a3f8c7b0b0fec7085e6 [sys.catalog]: This master's current role is: LEADER
I20260928 21:58:12.416742 23383 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:58:12.417232 23383 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:58:12.418773 23383 catalog_manager.cc:1347] Generated new cluster ID: 651e149052604d48977752debfbb40d3
I20260928 21:58:12.418799 23376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:12.407902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46829 (local address 127.0.0.1:43067)
0928 21:58:12.408067 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:12.408077 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:12.408112 (+ 35us) server_negotiation.cc:407] Connection header received
0928 21:58:12.408562 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:12.408571 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:12.408633 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:12.408781 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:12.409345 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:12.410742 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:12.411664 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:12.412242 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:12.412353 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:12.414726 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:12.414731 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:12.414731 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:12.414743 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:12.417080 (+ 2337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:12.417408 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:12.417411 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:12.417412 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:12.417522 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:12.417716 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:12.417717 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:12.417717 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:12.418026 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:12.418176 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:12.418433 (+ 257us) server_negotiation.cc:299] Negotiation successful
0928 21:58:12.418569 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":85,"threads_started":1}
I20260928 21:58:12.418824 23383 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:58:12.420262 2777 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:58:12.430056 23383 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:58:12.430794 23383 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:58:12.433678 23376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:12.427183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58103 (local address 127.0.0.1:43067)
0928 21:58:12.427243 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:12.427245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:12.427257 (+ 12us) server_negotiation.cc:407] Connection header received
0928 21:58:12.427283 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:12.427284 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:12.427295 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:12.427389 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:12.428092 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:12.428844 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:12.429896 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:12.430188 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:12.430247 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:12.431445 (+ 1198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:12.431446 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:12.431447 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:12.431454 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:12.432686 (+ 1232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:12.432988 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:12.432989 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:12.432990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:12.433044 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:12.433210 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:12.433211 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:12.433211 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:12.433319 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:12.433402 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:12.433502 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 21:58:12.433583 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 21:58:12.447551 23383 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29f23f8e09cf4a3f8c7b0b0fec7085e6: Generated new TSK 0
I20260928 21:58:12.447788 23383 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:58:12.450659 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 23305
2026-09-28T21:58:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1282 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632440055392-2777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:58:12 dist-test-slave-k6lz krb5kdc[23404](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:58:12 dist-test-slave-k6lz krb5kdc[23404](info): set up 2 sockets
Sep 28 21:58:12 dist-test-slave-k6lz krb5kdc[23404](info): commencing operation
krb5kdc: starting...
W20260928 21:58:13.411434 2777 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Sep 28 21:58:13 dist-test-slave-k6lz krb5kdc[23404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632693, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:58:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:58:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632440055392-2777-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632440055392-2777-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 21:58:13.530573 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632440055392-2777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632440055392-2777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632440055392-2777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43467
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632440055392-2777-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43467
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:58:13.612855 23417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:58:13.613082 23417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:58:13.613112 23417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:58:13.614378 23417 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:58:13.614430 23417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:58:13.614444 23417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:58:13.614454 23417 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:58:13.614465 23417 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:58:13.616484 23417 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632440055392-2777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632440055392-2777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632440055392-2777-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632440055392-2777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:58:13.616796 23417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:58:13.617065 23417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:58:13.623468 23422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:13.623653 23425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:13.623768 23423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:13.623910 23417 server_base.cc:1034] running on GCE node
I20260928 21:58:13.624178 23417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:58:13.624564 23417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:58:13.625746 23417 hybrid_clock.cc:648] HybridClock initialized: now 1790632693625715 us; error 54 us; skew 500 ppm
Sep 28 21:58:13 dist-test-slave-k6lz krb5kdc[23404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632693, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:58:13.628774 23417 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 21:58:13.629235 23417 webserver.cc:460] Webserver started at http://127.0.0.1:41155/ using document root <none> and password file <none>
I20260928 21:58:13.629472 23417 fs_manager.cc:359] Metadata directory not provided
I20260928 21:58:13.629519 23417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:58:13.629673 23417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:58:13.631165 23417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632440055392-2777-0/minicluster-data/master-0/data/instance:
uuid: "e46fdcd0f21c46a6887d29f389fd97ba"
format_stamp: "Formatted at 2026-09-28 21:58:13 on dist-test-slave-k6lz"
server_key: "4a5a45cc2e494d45c999e6ff83b81745"
server_key_iv: "8904c5ebfcac93a1ae616eaab332b6df"
server_key_version: "encryptionkey@0"
I20260928 21:58:13.631764 23417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632440055392-2777-0/minicluster-data/master-0/wal/instance:
uuid: "e46fdcd0f21c46a6887d29f389fd97ba"
format_stamp: "Formatted at 2026-09-28 21:58:13 on dist-test-slave-k6lz"
server_key: "4a5a45cc2e494d45c999e6ff83b81745"
server_key_iv: "8904c5ebfcac93a1ae616eaab332b6df"
server_key_version: "encryptionkey@0"
I20260928 21:58:13.633304 23417 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:58:13.634255 23432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:13.634438 23417 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:58:13.634594 23417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632440055392-2777-0/minicluster-data/master-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632440055392-2777-0/minicluster-data/master-0/wal
uuid: "e46fdcd0f21c46a6887d29f389fd97ba"
format_stamp: "Formatted at 2026-09-28 21:58:13 on dist-test-slave-k6lz"
server_key: "4a5a45cc2e494d45c999e6ff83b81745"
server_key_iv: "8904c5ebfcac93a1ae616eaab332b6df"
server_key_version: "encryptionkey@0"
I20260928 21:58:13.634666 23417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632440055392-2777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632440055392-2777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632440055392-2777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:58:13.642228 23417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:58:13.644352 23417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:58:13.644490 23417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:58:13.648519 23417 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43467
I20260928 21:58:13.648556 23484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43467 every 8 connection(s)
I20260928 21:58:13.648885 23417 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
I20260928 21:58:13.649887 23485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:58:13.652578 23485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e46fdcd0f21c46a6887d29f389fd97ba: Bootstrap starting.
I20260928 21:58:13.653158 23485 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e46fdcd0f21c46a6887d29f389fd97ba: Neither blocks nor log segments found. Creating new log.
I20260928 21:58:13.653409 23485 log.cc:824] T 00000000000000000000000000000000 P e46fdcd0f21c46a6887d29f389fd97ba: Log is configured to *not* fsync() on all Append() calls
I20260928 21:58:13.654166 23485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e46fdcd0f21c46a6887d29f389fd97ba: No bootstrap required, opened a new log
I20260928 21:58:13.655511 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 23417
I20260928 21:58:13.655618 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632440055392-2777-0/minicluster-data/master-0/wal/instance
I20260928 21:58:13.655774 2777 external_mini_cluster.cc:1428] Setting key 60706fe60463676fe3b3ccd5a9923d6f
I20260928 21:58:13.655931 23485 raft_consensus.cc:348] T 00000000000000000000000000000000 P e46fdcd0f21c46a6887d29f389fd97ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e46fdcd0f21c46a6887d29f389fd97ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43467 } }
I20260928 21:58:13.656131 23485 raft_consensus.cc:374] T 00000000000000000000000000000000 P e46fdcd0f21c46a6887d29f389fd97ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:58:13.656175 23485 raft_consensus.cc:729] T 00000000000000000000000000000000 P e46fdcd0f21c46a6887d29f389fd97ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e46fdcd0f21c46a6887d29f389fd97ba, State: Initialized, Role: FOLLOWER
I20260928 21:58:13.656287 23485 consensus_queue.cc:260] T 00000000000000000000000000000000 P e46fdcd0f21c46a6887d29f389fd97ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e46fdcd0f21c46a6887d29f389fd97ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43467 } }
I20260928 21:58:13.656368 23485 raft_consensus.cc:388] T 00000000000000000000000000000000 P e46fdcd0f21c46a6887d29f389fd97ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:58:13.656397 23485 raft_consensus.cc:482] T 00000000000000000000000000000000 P e46fdcd0f21c46a6887d29f389fd97ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:58:13.656445 23485 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e46fdcd0f21c46a6887d29f389fd97ba [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:58:13.657249 23485 raft_consensus.cc:504] T 00000000000000000000000000000000 P e46fdcd0f21c46a6887d29f389fd97ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e46fdcd0f21c46a6887d29f389fd97ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43467 } }
I20260928 21:58:13.657354 23485 leader_election.cc:302] T 00000000000000000000000000000000 P e46fdcd0f21c46a6887d29f389fd97ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e46fdcd0f21c46a6887d29f389fd97ba; no voters:
I20260928 21:58:13.657505 23485 leader_election.cc:288] T 00000000000000000000000000000000 P e46fdcd0f21c46a6887d29f389fd97ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:58:13.657549 23490 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e46fdcd0f21c46a6887d29f389fd97ba [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:58:13.657655 23490 raft_consensus.cc:686] T 00000000000000000000000000000000 P e46fdcd0f21c46a6887d29f389fd97ba [term 1 LEADER]: Becoming Leader. State: Replica: e46fdcd0f21c46a6887d29f389fd97ba, State: Running, Role: LEADER
I20260928 21:58:13.657765 23485 sys_catalog.cc:564] T 00000000000000000000000000000000 P e46fdcd0f21c46a6887d29f389fd97ba [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:58:13.657734 23490 consensus_queue.cc:237] T 00000000000000000000000000000000 P e46fdcd0f21c46a6887d29f389fd97ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e46fdcd0f21c46a6887d29f389fd97ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43467 } }
I20260928 21:58:13.658077 23490 sys_catalog.cc:455] T 00000000000000000000000000000000 P e46fdcd0f21c46a6887d29f389fd97ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader e46fdcd0f21c46a6887d29f389fd97ba. Latest consensus state: current_term: 1 leader_uuid: "e46fdcd0f21c46a6887d29f389fd97ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e46fdcd0f21c46a6887d29f389fd97ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43467 } } }
I20260928 21:58:13.658205 23490 sys_catalog.cc:458] T 00000000000000000000000000000000 P e46fdcd0f21c46a6887d29f389fd97ba [sys.catalog]: This master's current role is: LEADER
I20260928 21:58:13.658092 23492 sys_catalog.cc:455] T 00000000000000000000000000000000 P e46fdcd0f21c46a6887d29f389fd97ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e46fdcd0f21c46a6887d29f389fd97ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e46fdcd0f21c46a6887d29f389fd97ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43467 } } }
I20260928 21:58:13.658340 23492 sys_catalog.cc:458] T 00000000000000000000000000000000 P e46fdcd0f21c46a6887d29f389fd97ba [sys.catalog]: This master's current role is: LEADER
I20260928 21:58:13.658429 23497 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:58:13.658936 23497 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:58:13.660622 23497 catalog_manager.cc:1347] Generated new cluster ID: 25ac5489219a461cb302717102079df2
I20260928 21:58:13.660681 23497 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:58:13.661909 23489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:13.656376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52879 (local address 127.0.0.1:43467)
0928 21:58:13.656589 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:13.656599 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:13.656740 (+ 141us) server_negotiation.cc:407] Connection header received
0928 21:58:13.656910 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:13.656916 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:13.656961 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:13.657082 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:13.657285 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:13.658567 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:13.659682 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:13.660262 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:13.660373 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:13.661153 (+ 780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:13.661156 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:13.661157 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:58:13.661346 (+ 189us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:13.661377 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:58:13.661388 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:13.661444 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:13.661633 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 21:58:13.661702 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":94,"threads_started":1}
I20260928 21:58:13.662730 2777 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:58:13.666648 23489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:13.664038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45761 (local address 127.0.0.1:43467)
0928 21:58:13.664073 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:13.664075 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:13.664130 (+ 55us) server_negotiation.cc:407] Connection header received
0928 21:58:13.664193 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:13.664194 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:13.664204 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:13.664250 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:13.664371 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:13.664845 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:13.665539 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:13.665764 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:13.665819 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:13.666408 (+ 589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:13.666410 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:13.666410 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:58:13.666426 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:13.666441 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:58:13.666445 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:13.666491 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:13.666541 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 21:58:13.666566 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 21:58:13.674027 23497 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:58:13.674877 23497 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:58:13.679963 23497 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e46fdcd0f21c46a6887d29f389fd97ba: Generated new TSK 0
I20260928 21:58:13.680178 23497 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:58:13.685650 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 23417
2026-09-28T21:58:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1235 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632440055392-2777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:58:13 dist-test-slave-k6lz krb5kdc[23516](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:58:13 dist-test-slave-k6lz krb5kdc[23516](info): set up 2 sockets
Sep 28 21:58:13 dist-test-slave-k6lz krb5kdc[23516](info): commencing operation
krb5kdc: starting...
W20260928 21:58:14.651072 2777 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Sep 28 21:58:14 dist-test-slave-k6lz krb5kdc[23516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632694, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:58:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:58:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632440055392-2777-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632440055392-2777-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 21:58:14.768386 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632440055392-2777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632440055392-2777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632440055392-2777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45031
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632440055392-2777-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45031
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:58:14.848253 23529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:58:14.848471 23529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:58:14.848502 23529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:58:14.849681 23529 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:58:14.849722 23529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:58:14.849742 23529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:58:14.849758 23529 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:58:14.849776 23529 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:58:14.851265 23529 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632440055392-2777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632440055392-2777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45031
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632440055392-2777-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632440055392-2777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:58:14.851537 23529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:58:14.851744 23529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:58:14.856235 23534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:14.856235 23537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:14.856267 23535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:14.856375 23529 server_base.cc:1034] running on GCE node
I20260928 21:58:14.856679 23529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:58:14.856946 23529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:58:14.858090 23529 hybrid_clock.cc:648] HybridClock initialized: now 1790632694858069 us; error 33 us; skew 500 ppm
Sep 28 21:58:14 dist-test-slave-k6lz krb5kdc[23516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632694, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:58:14.867412 23529 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 21:58:14.867838 23529 webserver.cc:460] Webserver started at http://127.0.0.1:37249/ using document root <none> and password file <none>
I20260928 21:58:14.868078 23529 fs_manager.cc:359] Metadata directory not provided
I20260928 21:58:14.868129 23529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:58:14.868249 23529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:58:14.869287 23529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632440055392-2777-0/minicluster-data/master-0/data/instance:
uuid: "8fe85e68a85a43e5b76e7ec0b652eca5"
format_stamp: "Formatted at 2026-09-28 21:58:14 on dist-test-slave-k6lz"
server_key: "e827300c71e755de7ae0d894c744febc"
server_key_iv: "c06de03a014df2dc442c3df47dfbdda5"
server_key_version: "encryptionkey@0"
I20260928 21:58:14.869606 23529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632440055392-2777-0/minicluster-data/master-0/wal/instance:
uuid: "8fe85e68a85a43e5b76e7ec0b652eca5"
format_stamp: "Formatted at 2026-09-28 21:58:14 on dist-test-slave-k6lz"
server_key: "e827300c71e755de7ae0d894c744febc"
server_key_iv: "c06de03a014df2dc442c3df47dfbdda5"
server_key_version: "encryptionkey@0"
I20260928 21:58:14.870813 23529 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:58:14.871500 23544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:14.871677 23529 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:58:14.871778 23529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632440055392-2777-0/minicluster-data/master-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632440055392-2777-0/minicluster-data/master-0/wal
uuid: "8fe85e68a85a43e5b76e7ec0b652eca5"
format_stamp: "Formatted at 2026-09-28 21:58:14 on dist-test-slave-k6lz"
server_key: "e827300c71e755de7ae0d894c744febc"
server_key_iv: "c06de03a014df2dc442c3df47dfbdda5"
server_key_version: "encryptionkey@0"
I20260928 21:58:14.871843 23529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632440055392-2777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632440055392-2777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632440055392-2777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:58:14.875075 23529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:58:14.877010 23529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:58:14.877136 23529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:58:14.880875 23529 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45031
I20260928 21:58:14.880945 23596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45031 every 8 connection(s)
I20260928 21:58:14.881217 23529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
I20260928 21:58:14.882141 23597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:58:14.882848 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 23529
I20260928 21:58:14.882952 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632440055392-2777-0/minicluster-data/master-0/wal/instance
I20260928 21:58:14.883132 2777 external_mini_cluster.cc:1428] Setting key c20d1a265bcd7ff450caf2beed6ed496
I20260928 21:58:14.885109 23597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fe85e68a85a43e5b76e7ec0b652eca5: Bootstrap starting.
I20260928 21:58:14.885573 23600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:14.883641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41255 (local address 127.0.0.1:45031)
0928 21:58:14.883817 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:14.883826 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:14.883858 (+ 32us) server_negotiation.cc:407] Connection header received
0928 21:58:14.884338 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:14.884347 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:14.884382 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:14.884528 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:14.885153 (+ 625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:14.885157 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:14.885158 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:58:14.885162 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:14.885208 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:58:14.885222 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:14.885289 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:14.885346 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 21:58:14.885384 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":75,"threads_started":1}
I20260928 21:58:14.885795 23597 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8fe85e68a85a43e5b76e7ec0b652eca5: Neither blocks nor log segments found. Creating new log.
I20260928 21:58:14.886044 23597 log.cc:824] T 00000000000000000000000000000000 P 8fe85e68a85a43e5b76e7ec0b652eca5: Log is configured to *not* fsync() on all Append() calls
I20260928 21:58:14.886654 2777 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:58:14.886829 23597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fe85e68a85a43e5b76e7ec0b652eca5: No bootstrap required, opened a new log
I20260928 21:58:14.889096 23597 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8fe85e68a85a43e5b76e7ec0b652eca5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fe85e68a85a43e5b76e7ec0b652eca5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45031 } }
I20260928 21:58:14.889272 23597 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8fe85e68a85a43e5b76e7ec0b652eca5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:58:14.889331 23597 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8fe85e68a85a43e5b76e7ec0b652eca5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fe85e68a85a43e5b76e7ec0b652eca5, State: Initialized, Role: FOLLOWER
I20260928 21:58:14.889433 23600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:14.888256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48061 (local address 127.0.0.1:45031)
0928 21:58:14.888305 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:14.888307 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:14.888424 (+ 117us) server_negotiation.cc:407] Connection header received
0928 21:58:14.888468 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:14.888468 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:14.888473 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:14.888544 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:14.889153 (+ 609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:14.889155 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:14.889155 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:58:14.889159 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:14.889172 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:58:14.889176 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:14.889236 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:14.889270 (+ 34us) server_negotiation.cc:299] Negotiation successful
0928 21:58:14.889281 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 21:58:14.889453 23597 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8fe85e68a85a43e5b76e7ec0b652eca5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fe85e68a85a43e5b76e7ec0b652eca5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45031 } }
I20260928 21:58:14.889542 23597 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8fe85e68a85a43e5b76e7ec0b652eca5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:58:14.889575 23597 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8fe85e68a85a43e5b76e7ec0b652eca5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:58:14.889616 23597 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8fe85e68a85a43e5b76e7ec0b652eca5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:58:14.890205 23597 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8fe85e68a85a43e5b76e7ec0b652eca5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fe85e68a85a43e5b76e7ec0b652eca5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45031 } }
I20260928 21:58:14.890300 23597 leader_election.cc:302] T 00000000000000000000000000000000 P 8fe85e68a85a43e5b76e7ec0b652eca5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8fe85e68a85a43e5b76e7ec0b652eca5; no voters:
I20260928 21:58:14.890450 23597 leader_election.cc:288] T 00000000000000000000000000000000 P 8fe85e68a85a43e5b76e7ec0b652eca5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:58:14.890486 23607 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8fe85e68a85a43e5b76e7ec0b652eca5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:58:14.890635 23607 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8fe85e68a85a43e5b76e7ec0b652eca5 [term 1 LEADER]: Becoming Leader. State: Replica: 8fe85e68a85a43e5b76e7ec0b652eca5, State: Running, Role: LEADER
I20260928 21:58:14.890674 23597 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8fe85e68a85a43e5b76e7ec0b652eca5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:58:14.890745 23607 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8fe85e68a85a43e5b76e7ec0b652eca5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fe85e68a85a43e5b76e7ec0b652eca5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45031 } }
I20260928 21:58:14.891084 23608 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fe85e68a85a43e5b76e7ec0b652eca5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8fe85e68a85a43e5b76e7ec0b652eca5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fe85e68a85a43e5b76e7ec0b652eca5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45031 } } }
I20260928 21:58:14.891091 23609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fe85e68a85a43e5b76e7ec0b652eca5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8fe85e68a85a43e5b76e7ec0b652eca5. Latest consensus state: current_term: 1 leader_uuid: "8fe85e68a85a43e5b76e7ec0b652eca5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fe85e68a85a43e5b76e7ec0b652eca5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45031 } } }
I20260928 21:58:14.891230 23608 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fe85e68a85a43e5b76e7ec0b652eca5 [sys.catalog]: This master's current role is: LEADER
I20260928 21:58:14.891301 23609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fe85e68a85a43e5b76e7ec0b652eca5 [sys.catalog]: This master's current role is: LEADER
I20260928 21:58:14.891705 23616 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:58:14.892155 23616 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:58:14.893725 23616 catalog_manager.cc:1347] Generated new cluster ID: e9fca2129d5743a19f33085aa962e714
I20260928 21:58:14.893769 23616 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:58:14.903013 23616 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:58:14.903198 23616 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:58:14.914187 23616 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8fe85e68a85a43e5b76e7ec0b652eca5: Generated new TSK 0
I20260928 21:58:14.914422 23616 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:58:14.918684 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 23529
2026-09-28T21:58:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1232 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632440055392-2777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:58:14 dist-test-slave-k6lz krb5kdc[23628](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:58:14 dist-test-slave-k6lz krb5kdc[23628](info): set up 2 sockets
Sep 28 21:58:14 dist-test-slave-k6lz krb5kdc[23628](info): commencing operation
krb5kdc: starting...
W20260928 21:58:15.883313 2777 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Sep 28 21:58:15 dist-test-slave-k6lz krb5kdc[23628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632695, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:58:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:58:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632440055392-2777-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632440055392-2777-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 21:58:16.003197 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632440055392-2777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632440055392-2777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632440055392-2777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44463
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632440055392-2777-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44463
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:58:16.085597 23641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:58:16.085819 23641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:58:16.085843 23641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:58:16.086975 23641 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:58:16.087014 23641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:58:16.087026 23641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:58:16.087038 23641 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:58:16.087049 23641 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:58:16.088560 23641 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632440055392-2777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632440055392-2777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44463
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632440055392-2777-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632440055392-2777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:58:16.088766 23641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:58:16.088941 23641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:58:16.093753 23647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:16.093760 23646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:16.093777 23649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:16.093775 23641 server_base.cc:1034] running on GCE node
I20260928 21:58:16.094170 23641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:58:16.094455 23641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:58:16.095638 23641 hybrid_clock.cc:648] HybridClock initialized: now 1790632696095624 us; error 30 us; skew 500 ppm
Sep 28 21:58:16 dist-test-slave-k6lz krb5kdc[23628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632696, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:58:16.098043 23641 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 21:58:16.098443 23641 webserver.cc:460] Webserver started at http://127.0.0.1:43385/ using document root <none> and password file <none>
I20260928 21:58:16.098707 23641 fs_manager.cc:359] Metadata directory not provided
I20260928 21:58:16.098754 23641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:58:16.098842 23641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:58:16.099913 23641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632440055392-2777-0/minicluster-data/master-0/data/instance:
uuid: "67bc1addf9e342a6b78ea5307ccd9935"
format_stamp: "Formatted at 2026-09-28 21:58:16 on dist-test-slave-k6lz"
server_key: "05a88ff0600209596b09f5fa10c4fba6"
server_key_iv: "e0ecd9a3ada394988e088d38fcfb9dec"
server_key_version: "encryptionkey@0"
I20260928 21:58:16.100199 23641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632440055392-2777-0/minicluster-data/master-0/wal/instance:
uuid: "67bc1addf9e342a6b78ea5307ccd9935"
format_stamp: "Formatted at 2026-09-28 21:58:16 on dist-test-slave-k6lz"
server_key: "05a88ff0600209596b09f5fa10c4fba6"
server_key_iv: "e0ecd9a3ada394988e088d38fcfb9dec"
server_key_version: "encryptionkey@0"
I20260928 21:58:16.101409 23641 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:58:16.102276 23656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:16.102602 23641 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:58:16.102702 23641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632440055392-2777-0/minicluster-data/master-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632440055392-2777-0/minicluster-data/master-0/wal
uuid: "67bc1addf9e342a6b78ea5307ccd9935"
format_stamp: "Formatted at 2026-09-28 21:58:16 on dist-test-slave-k6lz"
server_key: "05a88ff0600209596b09f5fa10c4fba6"
server_key_iv: "e0ecd9a3ada394988e088d38fcfb9dec"
server_key_version: "encryptionkey@0"
I20260928 21:58:16.102764 23641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632440055392-2777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632440055392-2777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632440055392-2777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:58:16.117827 23641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:58:16.120081 23641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:58:16.120205 23641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:58:16.123999 23641 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44463
I20260928 21:58:16.124058 23708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44463 every 8 connection(s)
I20260928 21:58:16.124377 23641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
I20260928 21:58:16.125427 23709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:58:16.127857 23709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67bc1addf9e342a6b78ea5307ccd9935: Bootstrap starting.
I20260928 21:58:16.128211 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 23641
I20260928 21:58:16.128345 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632440055392-2777-0/minicluster-data/master-0/wal/instance
I20260928 21:58:16.128562 2777 external_mini_cluster.cc:1428] Setting key 2f82a5da4a2823734123dfd03aeed18c
I20260928 21:58:16.128644 23709 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67bc1addf9e342a6b78ea5307ccd9935: Neither blocks nor log segments found. Creating new log.
I20260928 21:58:16.128912 23709 log.cc:824] T 00000000000000000000000000000000 P 67bc1addf9e342a6b78ea5307ccd9935: Log is configured to *not* fsync() on all Append() calls
I20260928 21:58:16.129619 23709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67bc1addf9e342a6b78ea5307ccd9935: No bootstrap required, opened a new log
I20260928 21:58:16.131857 23709 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67bc1addf9e342a6b78ea5307ccd9935 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67bc1addf9e342a6b78ea5307ccd9935" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44463 } }
I20260928 21:58:16.132037 23709 raft_consensus.cc:374] T 00000000000000000000000000000000 P 67bc1addf9e342a6b78ea5307ccd9935 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:58:16.132074 23709 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67bc1addf9e342a6b78ea5307ccd9935 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67bc1addf9e342a6b78ea5307ccd9935, State: Initialized, Role: FOLLOWER
I20260928 21:58:16.132205 23709 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67bc1addf9e342a6b78ea5307ccd9935 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67bc1addf9e342a6b78ea5307ccd9935" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44463 } }
I20260928 21:58:16.132294 23709 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67bc1addf9e342a6b78ea5307ccd9935 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:58:16.132439 23709 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67bc1addf9e342a6b78ea5307ccd9935 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:58:16.132496 23709 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67bc1addf9e342a6b78ea5307ccd9935 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:58:16.133200 23709 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67bc1addf9e342a6b78ea5307ccd9935 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67bc1addf9e342a6b78ea5307ccd9935" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44463 } }
I20260928 21:58:16.133294 23709 leader_election.cc:302] T 00000000000000000000000000000000 P 67bc1addf9e342a6b78ea5307ccd9935 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67bc1addf9e342a6b78ea5307ccd9935; no voters:
I20260928 21:58:16.133441 23709 leader_election.cc:288] T 00000000000000000000000000000000 P 67bc1addf9e342a6b78ea5307ccd9935 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:58:16.133499 23714 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67bc1addf9e342a6b78ea5307ccd9935 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:58:16.133694 23709 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67bc1addf9e342a6b78ea5307ccd9935 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:58:16.133771 23714 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67bc1addf9e342a6b78ea5307ccd9935 [term 1 LEADER]: Becoming Leader. State: Replica: 67bc1addf9e342a6b78ea5307ccd9935, State: Running, Role: LEADER
I20260928 21:58:16.133879 23714 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67bc1addf9e342a6b78ea5307ccd9935 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67bc1addf9e342a6b78ea5307ccd9935" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44463 } }
I20260928 21:58:16.134238 23714 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67bc1addf9e342a6b78ea5307ccd9935 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67bc1addf9e342a6b78ea5307ccd9935. Latest consensus state: current_term: 1 leader_uuid: "67bc1addf9e342a6b78ea5307ccd9935" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67bc1addf9e342a6b78ea5307ccd9935" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44463 } } }
I20260928 21:58:16.134384 23714 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67bc1addf9e342a6b78ea5307ccd9935 [sys.catalog]: This master's current role is: LEADER
I20260928 21:58:16.135171 23715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67bc1addf9e342a6b78ea5307ccd9935 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "67bc1addf9e342a6b78ea5307ccd9935" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67bc1addf9e342a6b78ea5307ccd9935" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44463 } } }
I20260928 21:58:16.135284 23715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67bc1addf9e342a6b78ea5307ccd9935 [sys.catalog]: This master's current role is: LEADER
I20260928 21:58:16.135517 23719 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 28 21:58:16 dist-test-slave-k6lz krb5kdc[23628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632695, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 21:58:16.136088 23719 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:58:16.138036 23719 catalog_manager.cc:1347] Generated new cluster ID: 348b73915295465a8ab4a28388665f3a
I20260928 21:58:16.138096 23719 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:58:16.140329 23713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:16.129140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.102:51013 (local address 127.0.0.1:44463)
0928 21:58:16.129293 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:16.129301 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:16.129326 (+ 25us) server_negotiation.cc:407] Connection header received
0928 21:58:16.129793 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:16.129800 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:16.129854 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:16.129981 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:16.130880 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:16.131880 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:16.132987 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:16.133571 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:16.134083 (+ 512us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:16.136543 (+ 2460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:16.136547 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:16.136547 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:16.136560 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:16.138618 (+ 2058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:16.139070 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:16.139073 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:16.139073 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:16.139184 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:16.139401 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:16.139402 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:16.139402 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:16.139720 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:16.139906 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:16.140006 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 21:58:16.140109 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":65,"threads_started":1}
I20260928 21:58:16.141269 2777 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:58:16.145354 23719 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:58:16.146028 23719 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:58:16.149808 23713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:16.142656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.102:33391 (local address 127.0.0.1:44463)
0928 21:58:16.142692 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:16.142694 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:16.142956 (+ 262us) server_negotiation.cc:407] Connection header received
0928 21:58:16.142982 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:16.142982 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:16.142993 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:16.143055 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:16.143643 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:16.144568 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:16.145422 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:16.145719 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:16.145757 (+ 38us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:16.147185 (+ 1428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:16.147187 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:16.147187 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:16.147194 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:16.148692 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:16.149076 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:16.149077 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:16.149077 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:16.149139 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:16.149327 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:16.149328 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:16.149328 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:16.149452 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:16.149550 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:16.149651 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 21:58:16.149713 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":127,"server-negotiator.queue_time_us":17}
I20260928 21:58:16.155360 23719 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 67bc1addf9e342a6b78ea5307ccd9935: Generated new TSK 0
I20260928 21:58:16.155555 23719 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:58:16.158289 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 23641
W20260928 21:58:16.162233 23626 connection.cc:535] client connection to 127.0.0.1:44463 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44463): Connection reset by peer (error 104)
2026-09-28T21:58:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1239 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632440055392-2777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:58:16 dist-test-slave-k6lz krb5kdc[23740](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:58:16 dist-test-slave-k6lz krb5kdc[23740](info): set up 2 sockets
Sep 28 21:58:16 dist-test-slave-k6lz krb5kdc[23740](info): commencing operation
krb5kdc: starting...
W20260928 21:58:17.111279 2777 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Sep 28 21:58:17 dist-test-slave-k6lz krb5kdc[23740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632697, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:58:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:58:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632440055392-2777-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632440055392-2777-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 21:58:17.231403 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632440055392-2777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632440055392-2777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632440055392-2777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33999
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632440055392-2777-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33999
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:58:17.312829 23753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:58:17.313051 23753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:58:17.313074 23753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:58:17.314263 23753 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:58:17.314311 23753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:58:17.314325 23753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:58:17.314337 23753 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:58:17.314347 23753 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:58:17.316011 23753 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632440055392-2777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632440055392-2777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33999
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632440055392-2777-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23753
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632440055392-2777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:58:17.316228 23753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:58:17.316418 23753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:58:17.321497 23758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:17.321501 23759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:17.321587 23753 server_base.cc:1034] running on GCE node
W20260928 21:58:17.321568 23761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:17.322095 23753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:58:17.322387 23753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:58:17.323547 23753 hybrid_clock.cc:648] HybridClock initialized: now 1790632697323531 us; error 32 us; skew 500 ppm
Sep 28 21:58:17 dist-test-slave-k6lz krb5kdc[23740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632697, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:58:17.326241 23753 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 21:58:17.326730 23753 webserver.cc:460] Webserver started at http://127.0.0.1:42485/ using document root <none> and password file <none>
I20260928 21:58:17.326965 23753 fs_manager.cc:359] Metadata directory not provided
I20260928 21:58:17.327013 23753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:58:17.327137 23753 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:58:17.328230 23753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632440055392-2777-0/minicluster-data/master-0/data/instance:
uuid: "5ad0f9bb2db54a8689f20b120d627553"
format_stamp: "Formatted at 2026-09-28 21:58:17 on dist-test-slave-k6lz"
server_key: "41169648ef91753bc355037bac025f2e"
server_key_iv: "a5a88bc6bd41f2e1f2f85c9ba014cc68"
server_key_version: "encryptionkey@0"
I20260928 21:58:17.328572 23753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632440055392-2777-0/minicluster-data/master-0/wal/instance:
uuid: "5ad0f9bb2db54a8689f20b120d627553"
format_stamp: "Formatted at 2026-09-28 21:58:17 on dist-test-slave-k6lz"
server_key: "41169648ef91753bc355037bac025f2e"
server_key_iv: "a5a88bc6bd41f2e1f2f85c9ba014cc68"
server_key_version: "encryptionkey@0"
I20260928 21:58:17.329893 23753 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:58:17.330643 23768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:17.330817 23753 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:58:17.330901 23753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632440055392-2777-0/minicluster-data/master-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632440055392-2777-0/minicluster-data/master-0/wal
uuid: "5ad0f9bb2db54a8689f20b120d627553"
format_stamp: "Formatted at 2026-09-28 21:58:17 on dist-test-slave-k6lz"
server_key: "41169648ef91753bc355037bac025f2e"
server_key_iv: "a5a88bc6bd41f2e1f2f85c9ba014cc68"
server_key_version: "encryptionkey@0"
I20260928 21:58:17.330972 23753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632440055392-2777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632440055392-2777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632440055392-2777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:58:17.341662 23753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:58:17.343779 23753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:58:17.343916 23753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:58:17.347841 23753 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33999
I20260928 21:58:17.347935 23820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33999 every 8 connection(s)
I20260928 21:58:17.348237 23753 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
I20260928 21:58:17.349442 23821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:58:17.352102 23821 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ad0f9bb2db54a8689f20b120d627553: Bootstrap starting.
I20260928 21:58:17.352705 23821 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ad0f9bb2db54a8689f20b120d627553: Neither blocks nor log segments found. Creating new log.
I20260928 21:58:17.352978 23821 log.cc:824] T 00000000000000000000000000000000 P 5ad0f9bb2db54a8689f20b120d627553: Log is configured to *not* fsync() on all Append() calls
I20260928 21:58:17.353773 23821 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ad0f9bb2db54a8689f20b120d627553: No bootstrap required, opened a new log
I20260928 21:58:17.355481 23821 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ad0f9bb2db54a8689f20b120d627553 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ad0f9bb2db54a8689f20b120d627553" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33999 } }
I20260928 21:58:17.355628 23821 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ad0f9bb2db54a8689f20b120d627553 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:58:17.355649 23821 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ad0f9bb2db54a8689f20b120d627553 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ad0f9bb2db54a8689f20b120d627553, State: Initialized, Role: FOLLOWER
I20260928 21:58:17.355777 23821 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ad0f9bb2db54a8689f20b120d627553 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ad0f9bb2db54a8689f20b120d627553" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33999 } }
I20260928 21:58:17.355837 23821 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ad0f9bb2db54a8689f20b120d627553 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:58:17.355855 23821 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ad0f9bb2db54a8689f20b120d627553 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:58:17.355899 23821 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ad0f9bb2db54a8689f20b120d627553 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:58:17.356346 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 23753
I20260928 21:58:17.356458 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632440055392-2777-0/minicluster-data/master-0/wal/instance
I20260928 21:58:17.356442 23821 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ad0f9bb2db54a8689f20b120d627553 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ad0f9bb2db54a8689f20b120d627553" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33999 } }
I20260928 21:58:17.356518 23821 leader_election.cc:302] T 00000000000000000000000000000000 P 5ad0f9bb2db54a8689f20b120d627553 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ad0f9bb2db54a8689f20b120d627553; no voters:
I20260928 21:58:17.356645 2777 external_mini_cluster.cc:1428] Setting key 6b3cbc62c5bb5f11e97f295186287504
I20260928 21:58:17.356678 23821 leader_election.cc:288] T 00000000000000000000000000000000 P 5ad0f9bb2db54a8689f20b120d627553 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:58:17.356720 23824 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ad0f9bb2db54a8689f20b120d627553 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:58:17.356855 23824 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ad0f9bb2db54a8689f20b120d627553 [term 1 LEADER]: Becoming Leader. State: Replica: 5ad0f9bb2db54a8689f20b120d627553, State: Running, Role: LEADER
I20260928 21:58:17.356954 23824 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ad0f9bb2db54a8689f20b120d627553 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ad0f9bb2db54a8689f20b120d627553" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33999 } }
I20260928 21:58:17.357002 23821 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ad0f9bb2db54a8689f20b120d627553 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:58:17.357318 23826 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ad0f9bb2db54a8689f20b120d627553 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ad0f9bb2db54a8689f20b120d627553. Latest consensus state: current_term: 1 leader_uuid: "5ad0f9bb2db54a8689f20b120d627553" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ad0f9bb2db54a8689f20b120d627553" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33999 } } }
I20260928 21:58:17.357414 23826 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ad0f9bb2db54a8689f20b120d627553 [sys.catalog]: This master's current role is: LEADER
I20260928 21:58:17.357565 23825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ad0f9bb2db54a8689f20b120d627553 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ad0f9bb2db54a8689f20b120d627553" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ad0f9bb2db54a8689f20b120d627553" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33999 } } }
I20260928 21:58:17.357820 23825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ad0f9bb2db54a8689f20b120d627553 [sys.catalog]: This master's current role is: LEADER
I20260928 21:58:17.358054 23830 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:58:17.358618 23830 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:58:17.360306 23830 catalog_manager.cc:1347] Generated new cluster ID: 96ec4ad99d584f689079c0767e605603
I20260928 21:58:17.360360 23830 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:58:17.362648 23833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:17.357577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.102:60811 (local address 127.0.0.1:33999)
0928 21:58:17.357746 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:17.357756 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:17.357999 (+ 243us) server_negotiation.cc:407] Connection header received
0928 21:58:17.358104 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:17.358112 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:17.358175 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:17.358309 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:17.358659 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:17.359948 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:17.360849 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:17.361289 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:17.361384 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:17.362037 (+ 653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:17.362041 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:17.362041 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:58:17.362203 (+ 162us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:17.362237 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:58:17.362248 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:17.362316 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:17.362424 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 21:58:17.362454 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":87,"threads_started":1}
I20260928 21:58:17.363575 2777 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:58:17.370179 23833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:17.364856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.102:57947 (local address 127.0.0.1:33999)
0928 21:58:17.364898 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:17.364899 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:17.367579 (+ 2680us) server_negotiation.cc:407] Connection header received
0928 21:58:17.367605 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:17.367605 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:17.367618 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:17.367669 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:17.367822 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:17.368317 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:17.368997 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:17.369224 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:17.369280 (+ 56us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:17.369934 (+ 654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:17.369935 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:17.369936 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:58:17.369941 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:17.369957 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:58:17.369961 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:17.370009 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:17.370082 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 21:58:17.370094 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 21:58:17.393970 23830 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:58:17.394729 23830 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:58:17.397966 23830 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ad0f9bb2db54a8689f20b120d627553: Generated new TSK 0
I20260928 21:58:17.398164 23830 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:58:17.400776 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 23753
W20260928 21:58:17.404597 23738 connection.cc:535] client connection to 127.0.0.1:33999 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33999): Connection reset by peer (error 104)
2026-09-28T21:58:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1242 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632440055392-2777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:58:17 dist-test-slave-k6lz krb5kdc[23852](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:58:17 dist-test-slave-k6lz krb5kdc[23852](info): set up 2 sockets
Sep 28 21:58:17 dist-test-slave-k6lz krb5kdc[23852](info): commencing operation
krb5kdc: starting...
W20260928 21:58:18.363255 2777 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Sep 28 21:58:18 dist-test-slave-k6lz krb5kdc[23852](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632698, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:58:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:58:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632440055392-2777-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632440055392-2777-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 21:58:18.480384 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632440055392-2777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632440055392-2777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632440055392-2777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35653
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632440055392-2777-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35653
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:58:18.559774 23865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:58:18.559971 23865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:58:18.559993 23865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:58:18.561131 23865 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:58:18.561177 23865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:58:18.561190 23865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:58:18.561201 23865 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:58:18.561213 23865 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:58:18.562726 23865 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632440055392-2777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632440055392-2777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35653
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632440055392-2777-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632440055392-2777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:58:18.562980 23865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:58:18.563230 23865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:58:18.568010 23873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:18.568182 23871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:18.568027 23870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:18.568331 23865 server_base.cc:1034] running on GCE node
I20260928 21:58:18.568557 23865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:58:18.568818 23865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:58:18.569976 23865 hybrid_clock.cc:648] HybridClock initialized: now 1790632698569955 us; error 31 us; skew 500 ppm
Sep 28 21:58:18 dist-test-slave-k6lz krb5kdc[23852](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632698, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:58:18.572432 23865 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 21:58:18.572867 23865 webserver.cc:460] Webserver started at http://127.0.0.1:46779/ using document root <none> and password file <none>
I20260928 21:58:18.573077 23865 fs_manager.cc:359] Metadata directory not provided
I20260928 21:58:18.573122 23865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:58:18.573241 23865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:58:18.574270 23865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632440055392-2777-0/minicluster-data/master-0/data/instance:
uuid: "606512bd25244140b0e4398b7f0000ff"
format_stamp: "Formatted at 2026-09-28 21:58:18 on dist-test-slave-k6lz"
server_key: "e2448d527bd8b109c7cd9af5c051ef98"
server_key_iv: "c4f4c8a8bc708a7fe41e9ab429323d8f"
server_key_version: "encryptionkey@0"
I20260928 21:58:18.574604 23865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632440055392-2777-0/minicluster-data/master-0/wal/instance:
uuid: "606512bd25244140b0e4398b7f0000ff"
format_stamp: "Formatted at 2026-09-28 21:58:18 on dist-test-slave-k6lz"
server_key: "e2448d527bd8b109c7cd9af5c051ef98"
server_key_iv: "c4f4c8a8bc708a7fe41e9ab429323d8f"
server_key_version: "encryptionkey@0"
I20260928 21:58:18.575845 23865 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 21:58:18.576561 23880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:18.576715 23865 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:18.576793 23865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632440055392-2777-0/minicluster-data/master-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632440055392-2777-0/minicluster-data/master-0/wal
uuid: "606512bd25244140b0e4398b7f0000ff"
format_stamp: "Formatted at 2026-09-28 21:58:18 on dist-test-slave-k6lz"
server_key: "e2448d527bd8b109c7cd9af5c051ef98"
server_key_iv: "c4f4c8a8bc708a7fe41e9ab429323d8f"
server_key_version: "encryptionkey@0"
I20260928 21:58:18.576854 23865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632440055392-2777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632440055392-2777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632440055392-2777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:58:18.579970 23865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:58:18.581894 23865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:58:18.582013 23865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:58:18.585611 23865 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35653
I20260928 21:58:18.585639 23932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35653 every 8 connection(s)
I20260928 21:58:18.585917 23865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
I20260928 21:58:18.586872 23933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:58:18.589056 23933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 606512bd25244140b0e4398b7f0000ff: Bootstrap starting.
I20260928 21:58:18.589563 23933 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 606512bd25244140b0e4398b7f0000ff: Neither blocks nor log segments found. Creating new log.
I20260928 21:58:18.589797 23933 log.cc:824] T 00000000000000000000000000000000 P 606512bd25244140b0e4398b7f0000ff: Log is configured to *not* fsync() on all Append() calls
I20260928 21:58:18.590412 23933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 606512bd25244140b0e4398b7f0000ff: No bootstrap required, opened a new log
I20260928 21:58:18.592114 23933 raft_consensus.cc:348] T 00000000000000000000000000000000 P 606512bd25244140b0e4398b7f0000ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "606512bd25244140b0e4398b7f0000ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35653 } }
I20260928 21:58:18.592250 23933 raft_consensus.cc:374] T 00000000000000000000000000000000 P 606512bd25244140b0e4398b7f0000ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:58:18.592271 23933 raft_consensus.cc:729] T 00000000000000000000000000000000 P 606512bd25244140b0e4398b7f0000ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 606512bd25244140b0e4398b7f0000ff, State: Initialized, Role: FOLLOWER
I20260928 21:58:18.592375 23933 consensus_queue.cc:260] T 00000000000000000000000000000000 P 606512bd25244140b0e4398b7f0000ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "606512bd25244140b0e4398b7f0000ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35653 } }
I20260928 21:58:18.592425 23933 raft_consensus.cc:388] T 00000000000000000000000000000000 P 606512bd25244140b0e4398b7f0000ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:58:18.592444 23933 raft_consensus.cc:482] T 00000000000000000000000000000000 P 606512bd25244140b0e4398b7f0000ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:58:18.592475 23933 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 606512bd25244140b0e4398b7f0000ff [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:58:18.592933 23933 raft_consensus.cc:504] T 00000000000000000000000000000000 P 606512bd25244140b0e4398b7f0000ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "606512bd25244140b0e4398b7f0000ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35653 } }
I20260928 21:58:18.593014 23933 leader_election.cc:302] T 00000000000000000000000000000000 P 606512bd25244140b0e4398b7f0000ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 606512bd25244140b0e4398b7f0000ff; no voters:
I20260928 21:58:18.593133 23933 leader_election.cc:288] T 00000000000000000000000000000000 P 606512bd25244140b0e4398b7f0000ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:58:18.593181 23936 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 606512bd25244140b0e4398b7f0000ff [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:58:18.593344 23933 sys_catalog.cc:564] T 00000000000000000000000000000000 P 606512bd25244140b0e4398b7f0000ff [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:58:18.593351 23936 raft_consensus.cc:686] T 00000000000000000000000000000000 P 606512bd25244140b0e4398b7f0000ff [term 1 LEADER]: Becoming Leader. State: Replica: 606512bd25244140b0e4398b7f0000ff, State: Running, Role: LEADER
I20260928 21:58:18.593497 23936 consensus_queue.cc:237] T 00000000000000000000000000000000 P 606512bd25244140b0e4398b7f0000ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "606512bd25244140b0e4398b7f0000ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35653 } }
I20260928 21:58:18.593845 23938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 606512bd25244140b0e4398b7f0000ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 606512bd25244140b0e4398b7f0000ff. Latest consensus state: current_term: 1 leader_uuid: "606512bd25244140b0e4398b7f0000ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "606512bd25244140b0e4398b7f0000ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35653 } } }
I20260928 21:58:18.593983 23938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 606512bd25244140b0e4398b7f0000ff [sys.catalog]: This master's current role is: LEADER
I20260928 21:58:18.593854 23937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 606512bd25244140b0e4398b7f0000ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "606512bd25244140b0e4398b7f0000ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "606512bd25244140b0e4398b7f0000ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35653 } } }
I20260928 21:58:18.594173 23940 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:58:18.594763 23940 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:58:18.594887 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 23865
I20260928 21:58:18.595013 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632440055392-2777-0/minicluster-data/master-0/wal/instance
I20260928 21:58:18.595191 2777 external_mini_cluster.cc:1428] Setting key c86ea77851f29b23ede7b0dfea7bc5b2
I20260928 21:58:18.594202 23937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 606512bd25244140b0e4398b7f0000ff [sys.catalog]: This master's current role is: LEADER
I20260928 21:58:18.596534 23940 catalog_manager.cc:1347] Generated new cluster ID: 38cf02f4519842349b8b389d451c9157
I20260928 21:58:18.596593 23940 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:58:18.600999 23955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:18.595751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.102:42529 (local address 127.0.0.1:35653)
0928 21:58:18.595929 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:18.595939 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:18.599706 (+ 3767us) server_negotiation.cc:407] Connection header received
0928 21:58:18.599828 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:18.599834 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:18.599857 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:18.599979 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:18.600635 (+ 656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:18.600638 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:18.600639 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:58:18.600642 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:18.600677 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:58:18.600688 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:18.600738 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:18.600808 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 21:58:18.600852 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":89,"threads_started":1}
I20260928 21:58:18.601927 2777 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:58:18.604291 23955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:18.603188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.102:48053 (local address 127.0.0.1:35653)
0928 21:58:18.603220 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:18.603222 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:18.603481 (+ 259us) server_negotiation.cc:407] Connection header received
0928 21:58:18.603544 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:18.603545 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:18.603550 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:18.603593 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:18.604086 (+ 493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:18.604087 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:18.604087 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:58:18.604091 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:18.604104 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:58:18.604108 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:18.604160 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:18.604220 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 21:58:18.604231 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 21:58:18.606834 23940 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:58:18.606998 23940 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:58:18.612913 23940 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 606512bd25244140b0e4398b7f0000ff: Generated new TSK 0
I20260928 21:58:18.613080 23940 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:58:18.614872 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 23865
2026-09-28T21:58:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1213 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632440055392-2777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:58:18 dist-test-slave-k6lz krb5kdc[23964](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:58:18 dist-test-slave-k6lz krb5kdc[23964](info): set up 2 sockets
Sep 28 21:58:18 dist-test-slave-k6lz krb5kdc[23964](info): commencing operation
krb5kdc: starting...
W20260928 21:58:19.583245 2777 mini_kdc.cc:121] Time spent starting KDC: real 0.951s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Sep 28 21:58:19 dist-test-slave-k6lz krb5kdc[23964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632699, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:58:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:58:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632440055392-2777-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632440055392-2777-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 21:58:19.702307 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632440055392-2777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632440055392-2777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632440055392-2777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33825
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632440055392-2777-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:58:19.781199 23977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:58:19.781415 23977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:58:19.781437 23977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:58:19.782588 23977 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:58:19.782630 23977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:58:19.782644 23977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:58:19.782653 23977 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:58:19.782665 23977 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:58:19.784111 23977 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632440055392-2777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632440055392-2777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33825
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632440055392-2777-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632440055392-2777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:58:19.784330 23977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:58:19.784538 23977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:58:19.789418 23977 server_base.cc:1034] running on GCE node
W20260928 21:58:19.789384 23983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:19.789484 23982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:19.789480 23985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:19.789866 23977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:58:19.790119 23977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:58:19.791255 23977 hybrid_clock.cc:648] HybridClock initialized: now 1790632699791245 us; error 27 us; skew 500 ppm
Sep 28 21:58:19 dist-test-slave-k6lz krb5kdc[23964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632699, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:58:19.793495 23977 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 21:58:19.793996 23977 webserver.cc:460] Webserver started at http://127.0.0.1:42237/ using document root <none> and password file <none>
I20260928 21:58:19.794206 23977 fs_manager.cc:359] Metadata directory not provided
I20260928 21:58:19.794252 23977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:58:19.794373 23977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:58:19.795507 23977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632440055392-2777-0/minicluster-data/master-0/data/instance:
uuid: "7937e6e20ecc45eeb0e2a3d7cc3a6252"
format_stamp: "Formatted at 2026-09-28 21:58:19 on dist-test-slave-k6lz"
server_key: "b088b3fb2feee89f02b447dc7e782746"
server_key_iv: "1b8f6011d1a1118ffb47c844a2dca445"
server_key_version: "encryptionkey@0"
I20260928 21:58:19.795823 23977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632440055392-2777-0/minicluster-data/master-0/wal/instance:
uuid: "7937e6e20ecc45eeb0e2a3d7cc3a6252"
format_stamp: "Formatted at 2026-09-28 21:58:19 on dist-test-slave-k6lz"
server_key: "b088b3fb2feee89f02b447dc7e782746"
server_key_iv: "1b8f6011d1a1118ffb47c844a2dca445"
server_key_version: "encryptionkey@0"
I20260928 21:58:19.797062 23977 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:58:19.797803 23992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:19.797977 23977 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:58:19.798058 23977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632440055392-2777-0/minicluster-data/master-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632440055392-2777-0/minicluster-data/master-0/wal
uuid: "7937e6e20ecc45eeb0e2a3d7cc3a6252"
format_stamp: "Formatted at 2026-09-28 21:58:19 on dist-test-slave-k6lz"
server_key: "b088b3fb2feee89f02b447dc7e782746"
server_key_iv: "1b8f6011d1a1118ffb47c844a2dca445"
server_key_version: "encryptionkey@0"
I20260928 21:58:19.798142 23977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632440055392-2777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632440055392-2777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632440055392-2777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:58:19.808225 23977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:58:19.810060 23977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:58:19.810194 23977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:58:19.814087 23977 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33825
I20260928 21:58:19.814133 24044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33825 every 8 connection(s)
I20260928 21:58:19.814467 23977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
I20260928 21:58:19.815416 24045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:58:19.816773 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 23977
I20260928 21:58:19.816879 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632440055392-2777-0/minicluster-data/master-0/wal/instance
I20260928 21:58:19.817068 2777 external_mini_cluster.cc:1428] Setting key 9aa299d105c4c2b5289e6df654520d6c
I20260928 21:58:19.817770 24045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7937e6e20ecc45eeb0e2a3d7cc3a6252: Bootstrap starting.
I20260928 21:58:19.818315 24045 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7937e6e20ecc45eeb0e2a3d7cc3a6252: Neither blocks nor log segments found. Creating new log.
I20260928 21:58:19.818598 24045 log.cc:824] T 00000000000000000000000000000000 P 7937e6e20ecc45eeb0e2a3d7cc3a6252: Log is configured to *not* fsync() on all Append() calls
I20260928 21:58:19.819289 24045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7937e6e20ecc45eeb0e2a3d7cc3a6252: No bootstrap required, opened a new log
I20260928 21:58:19.821138 24045 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7937e6e20ecc45eeb0e2a3d7cc3a6252 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7937e6e20ecc45eeb0e2a3d7cc3a6252" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33825 } }
I20260928 21:58:19.821274 24045 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7937e6e20ecc45eeb0e2a3d7cc3a6252 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:58:19.821305 24045 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7937e6e20ecc45eeb0e2a3d7cc3a6252 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7937e6e20ecc45eeb0e2a3d7cc3a6252, State: Initialized, Role: FOLLOWER
I20260928 21:58:19.821462 24045 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7937e6e20ecc45eeb0e2a3d7cc3a6252 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7937e6e20ecc45eeb0e2a3d7cc3a6252" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33825 } }
I20260928 21:58:19.821533 24045 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7937e6e20ecc45eeb0e2a3d7cc3a6252 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:58:19.821564 24045 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7937e6e20ecc45eeb0e2a3d7cc3a6252 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:58:19.821619 24045 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7937e6e20ecc45eeb0e2a3d7cc3a6252 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:58:19.822223 24045 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7937e6e20ecc45eeb0e2a3d7cc3a6252 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7937e6e20ecc45eeb0e2a3d7cc3a6252" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33825 } }
I20260928 21:58:19.822321 24045 leader_election.cc:302] T 00000000000000000000000000000000 P 7937e6e20ecc45eeb0e2a3d7cc3a6252 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7937e6e20ecc45eeb0e2a3d7cc3a6252; no voters:
I20260928 21:58:19.822474 24045 leader_election.cc:288] T 00000000000000000000000000000000 P 7937e6e20ecc45eeb0e2a3d7cc3a6252 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:58:19.822508 24050 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7937e6e20ecc45eeb0e2a3d7cc3a6252 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:58:19.822618 24050 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7937e6e20ecc45eeb0e2a3d7cc3a6252 [term 1 LEADER]: Becoming Leader. State: Replica: 7937e6e20ecc45eeb0e2a3d7cc3a6252, State: Running, Role: LEADER
I20260928 21:58:19.822754 24050 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7937e6e20ecc45eeb0e2a3d7cc3a6252 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7937e6e20ecc45eeb0e2a3d7cc3a6252" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33825 } }
I20260928 21:58:19.822803 24045 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7937e6e20ecc45eeb0e2a3d7cc3a6252 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:58:19.823104 24051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7937e6e20ecc45eeb0e2a3d7cc3a6252 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7937e6e20ecc45eeb0e2a3d7cc3a6252" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7937e6e20ecc45eeb0e2a3d7cc3a6252" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33825 } } }
I20260928 21:58:19.823172 24051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7937e6e20ecc45eeb0e2a3d7cc3a6252 [sys.catalog]: This master's current role is: LEADER
I20260928 21:58:19.823405 24052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7937e6e20ecc45eeb0e2a3d7cc3a6252 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7937e6e20ecc45eeb0e2a3d7cc3a6252. Latest consensus state: current_term: 1 leader_uuid: "7937e6e20ecc45eeb0e2a3d7cc3a6252" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7937e6e20ecc45eeb0e2a3d7cc3a6252" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33825 } } }
Sep 28 21:58:19 dist-test-slave-k6lz krb5kdc[23964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632699, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 21:58:19.823714 24052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7937e6e20ecc45eeb0e2a3d7cc3a6252 [sys.catalog]: This master's current role is: LEADER
I20260928 21:58:19.824105 24055 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:58:19.824810 24055 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:58:19.826494 24055 catalog_manager.cc:1347] Generated new cluster ID: 77f5bf44b7a14e339bbb359d110b6523
I20260928 21:58:19.826543 24055 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:58:19.827358 24048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:19.817561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.102:48451 (local address 127.0.0.1:33825)
0928 21:58:19.817759 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:19.817767 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:19.817792 (+ 25us) server_negotiation.cc:407] Connection header received
0928 21:58:19.818268 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:19.818277 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:19.818328 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:19.818456 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:19.819199 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:19.820123 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:19.820883 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:19.821292 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:19.821386 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:19.824087 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:19.824091 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:19.824092 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:19.824104 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:19.825764 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:19.826149 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:19.826152 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:19.826152 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:19.826237 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:19.826430 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:19.826431 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:19.826431 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:19.826725 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:19.826850 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:19.827043 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 21:58:19.827139 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":101,"threads_started":1}
I20260928 21:58:19.828285 2777 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:58:19.836520 24048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:19.829519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.102:53405 (local address 127.0.0.1:33825)
0928 21:58:19.829555 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:19.829556 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:19.829628 (+ 72us) server_negotiation.cc:407] Connection header received
0928 21:58:19.829688 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:19.829689 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:19.829698 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:19.829746 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:19.830356 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:19.831100 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:19.832140 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:19.832487 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:19.832545 (+ 58us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:19.833909 (+ 1364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:19.833910 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:19.833910 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:19.833916 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:19.835401 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:19.835801 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:19.835802 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:19.835803 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:19.835855 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:19.836051 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:19.836052 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:19.836052 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:19.836164 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:19.836247 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:19.836376 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 21:58:19.836430 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 21:58:19.838855 24055 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:58:19.839584 24055 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:58:19.846375 24055 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7937e6e20ecc45eeb0e2a3d7cc3a6252: Generated new TSK 0
I20260928 21:58:19.846580 24055 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:58:19.851145 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 23977
W20260928 21:58:19.854802 23962 connection.cc:535] client connection to 127.0.0.1:33825 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33825): Connection reset by peer (error 104)
2026-09-28T21:58:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1236 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632440055392-2777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:58:19 dist-test-slave-k6lz krb5kdc[24076](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:58:19 dist-test-slave-k6lz krb5kdc[24076](info): set up 2 sockets
Sep 28 21:58:19 dist-test-slave-k6lz krb5kdc[24076](info): commencing operation
krb5kdc: starting...
W20260928 21:58:20.815212 2777 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Sep 28 21:58:20 dist-test-slave-k6lz krb5kdc[24076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632700, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:58:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:58:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632440055392-2777-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632440055392-2777-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 21:58:20.934859 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632440055392-2777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632440055392-2777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632440055392-2777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40751
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632440055392-2777-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40751
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:58:21.015715 24089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:58:21.015937 24089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:58:21.015970 24089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:58:21.017174 24089 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:58:21.017220 24089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:58:21.017237 24089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:58:21.017252 24089 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:58:21.017268 24089 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:58:21.018833 24089 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632440055392-2777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632440055392-2777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40751
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632440055392-2777-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24089
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632440055392-2777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:58:21.019079 24089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:58:21.019291 24089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:58:21.024169 24095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:21.024264 24089 server_base.cc:1034] running on GCE node
W20260928 21:58:21.024169 24097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:21.024287 24094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:21.024716 24089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:58:21.024998 24089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:58:21.026146 24089 hybrid_clock.cc:648] HybridClock initialized: now 1790632701026112 us; error 49 us; skew 500 ppm
Sep 28 21:58:21 dist-test-slave-k6lz krb5kdc[24076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632701, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:58:21.028734 24089 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 21:58:21.029187 24089 webserver.cc:460] Webserver started at http://127.0.0.1:34951/ using document root <none> and password file <none>
I20260928 21:58:21.029407 24089 fs_manager.cc:359] Metadata directory not provided
I20260928 21:58:21.029449 24089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:58:21.029553 24089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:58:21.030579 24089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632440055392-2777-0/minicluster-data/master-0/data/instance:
uuid: "67a5ca6a3db942758fca1e4e77fe4cc6"
format_stamp: "Formatted at 2026-09-28 21:58:21 on dist-test-slave-k6lz"
server_key: "2184fb66d014fb2db346efe997b65da4"
server_key_iv: "6f54b4d55e85c657e0902485c4109efe"
server_key_version: "encryptionkey@0"
I20260928 21:58:21.030869 24089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632440055392-2777-0/minicluster-data/master-0/wal/instance:
uuid: "67a5ca6a3db942758fca1e4e77fe4cc6"
format_stamp: "Formatted at 2026-09-28 21:58:21 on dist-test-slave-k6lz"
server_key: "2184fb66d014fb2db346efe997b65da4"
server_key_iv: "6f54b4d55e85c657e0902485c4109efe"
server_key_version: "encryptionkey@0"
I20260928 21:58:21.032094 24089 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:58:21.032822 24104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:21.032970 24089 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:58:21.033054 24089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632440055392-2777-0/minicluster-data/master-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632440055392-2777-0/minicluster-data/master-0/wal
uuid: "67a5ca6a3db942758fca1e4e77fe4cc6"
format_stamp: "Formatted at 2026-09-28 21:58:21 on dist-test-slave-k6lz"
server_key: "2184fb66d014fb2db346efe997b65da4"
server_key_iv: "6f54b4d55e85c657e0902485c4109efe"
server_key_version: "encryptionkey@0"
I20260928 21:58:21.033119 24089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632440055392-2777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632440055392-2777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632440055392-2777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:58:21.053963 24089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:58:21.056267 24089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:58:21.056384 24089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:58:21.060375 24089 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40751
I20260928 21:58:21.060413 24156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40751 every 8 connection(s)
I20260928 21:58:21.060756 24089 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
I20260928 21:58:21.061645 24157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:58:21.063956 24157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67a5ca6a3db942758fca1e4e77fe4cc6: Bootstrap starting.
I20260928 21:58:21.064512 24157 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67a5ca6a3db942758fca1e4e77fe4cc6: Neither blocks nor log segments found. Creating new log.
I20260928 21:58:21.064770 24157 log.cc:824] T 00000000000000000000000000000000 P 67a5ca6a3db942758fca1e4e77fe4cc6: Log is configured to *not* fsync() on all Append() calls
I20260928 21:58:21.065526 24157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67a5ca6a3db942758fca1e4e77fe4cc6: No bootstrap required, opened a new log
I20260928 21:58:21.067099 24157 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67a5ca6a3db942758fca1e4e77fe4cc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67a5ca6a3db942758fca1e4e77fe4cc6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40751 } }
I20260928 21:58:21.067236 24157 raft_consensus.cc:374] T 00000000000000000000000000000000 P 67a5ca6a3db942758fca1e4e77fe4cc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:58:21.067260 24157 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67a5ca6a3db942758fca1e4e77fe4cc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67a5ca6a3db942758fca1e4e77fe4cc6, State: Initialized, Role: FOLLOWER
I20260928 21:58:21.067412 24157 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67a5ca6a3db942758fca1e4e77fe4cc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67a5ca6a3db942758fca1e4e77fe4cc6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40751 } }
I20260928 21:58:21.067478 24157 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67a5ca6a3db942758fca1e4e77fe4cc6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:58:21.067500 24157 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67a5ca6a3db942758fca1e4e77fe4cc6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:58:21.067546 24157 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67a5ca6a3db942758fca1e4e77fe4cc6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:58:21.068058 24157 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67a5ca6a3db942758fca1e4e77fe4cc6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67a5ca6a3db942758fca1e4e77fe4cc6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40751 } }
I20260928 21:58:21.068145 24157 leader_election.cc:302] T 00000000000000000000000000000000 P 67a5ca6a3db942758fca1e4e77fe4cc6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67a5ca6a3db942758fca1e4e77fe4cc6; no voters:
I20260928 21:58:21.068295 24157 leader_election.cc:288] T 00000000000000000000000000000000 P 67a5ca6a3db942758fca1e4e77fe4cc6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:58:21.068336 24160 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67a5ca6a3db942758fca1e4e77fe4cc6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:58:21.068497 24160 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67a5ca6a3db942758fca1e4e77fe4cc6 [term 1 LEADER]: Becoming Leader. State: Replica: 67a5ca6a3db942758fca1e4e77fe4cc6, State: Running, Role: LEADER
I20260928 21:58:21.068531 24157 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67a5ca6a3db942758fca1e4e77fe4cc6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:58:21.068645 24160 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67a5ca6a3db942758fca1e4e77fe4cc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67a5ca6a3db942758fca1e4e77fe4cc6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40751 } }
I20260928 21:58:21.069011 24162 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67a5ca6a3db942758fca1e4e77fe4cc6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67a5ca6a3db942758fca1e4e77fe4cc6. Latest consensus state: current_term: 1 leader_uuid: "67a5ca6a3db942758fca1e4e77fe4cc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67a5ca6a3db942758fca1e4e77fe4cc6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40751 } } }
I20260928 21:58:21.069139 24162 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67a5ca6a3db942758fca1e4e77fe4cc6 [sys.catalog]: This master's current role is: LEADER
I20260928 21:58:21.069159 24161 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67a5ca6a3db942758fca1e4e77fe4cc6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "67a5ca6a3db942758fca1e4e77fe4cc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67a5ca6a3db942758fca1e4e77fe4cc6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40751 } } }
I20260928 21:58:21.069305 24161 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67a5ca6a3db942758fca1e4e77fe4cc6 [sys.catalog]: This master's current role is: LEADER
I20260928 21:58:21.069521 24169 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:58:21.069940 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 24089
I20260928 21:58:21.070055 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632440055392-2777-0/minicluster-data/master-0/wal/instance
I20260928 21:58:21.070051 24169 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:58:21.070233 2777 external_mini_cluster.cc:1428] Setting key 0baed14cfa3ed107996cc5c3bd9c778e
I20260928 21:58:21.072046 24169 catalog_manager.cc:1347] Generated new cluster ID: 3f3c05d3ec61482a98da41587b509995
I20260928 21:58:21.072099 24169 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:58:21.076637 24179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:21.070677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.102:46603 (local address 127.0.0.1:40751)
0928 21:58:21.071109 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:21.071119 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:21.071293 (+ 174us) server_negotiation.cc:407] Connection header received
0928 21:58:21.071447 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:21.071452 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:21.071503 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:21.071615 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:21.071840 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:21.073271 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:21.074377 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:21.075032 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:21.075153 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:21.075892 (+ 739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:21.075897 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:21.075898 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:58:21.076120 (+ 222us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:21.076163 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:58:21.076179 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:21.076274 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:21.076372 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 21:58:21.076405 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":64,"threads_started":1}
I20260928 21:58:21.077644 2777 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:58:21.082551 24179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:21.079196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.102:43787 (local address 127.0.0.1:40751)
0928 21:58:21.079231 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:21.079233 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:21.079269 (+ 36us) server_negotiation.cc:407] Connection header received
0928 21:58:21.079324 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:21.079324 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:21.079357 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:21.079421 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:21.079552 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:21.080311 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:21.081269 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:21.081581 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:21.081648 (+ 67us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:21.082307 (+ 659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:21.082309 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:21.082309 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:58:21.082314 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:21.082330 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:58:21.082335 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:21.082386 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:21.082456 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 21:58:21.082468 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 21:58:21.086918 24169 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:58:21.087694 24169 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:58:21.090847 24169 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 67a5ca6a3db942758fca1e4e77fe4cc6: Generated new TSK 0
I20260928 21:58:21.091025 24169 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:58:21.095285 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 24089
W20260928 21:58:21.098948 24074 connection.cc:535] client connection to 127.0.0.1:40751 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40751): Connection reset by peer (error 104)
2026-09-28T21:58:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1244 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632440055392-2777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:58:21 dist-test-slave-k6lz krb5kdc[24188](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:58:21 dist-test-slave-k6lz krb5kdc[24188](info): set up 2 sockets
Sep 28 21:58:21 dist-test-slave-k6lz krb5kdc[24188](info): commencing operation
krb5kdc: starting...
W20260928 21:58:22.067286 2777 mini_kdc.cc:121] Time spent starting KDC: real 0.955s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Sep 28 21:58:22 dist-test-slave-k6lz krb5kdc[24188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:58:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:58:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632440055392-2777-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632440055392-2777-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 21:58:22.187238 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632440055392-2777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632440055392-2777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632440055392-2777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44967
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632440055392-2777-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44967
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:58:22.266640 24201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:58:22.266855 24201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:58:22.266877 24201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:58:22.268121 24201 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:58:22.268173 24201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:58:22.268186 24201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:58:22.268198 24201 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:58:22.268208 24201 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:58:22.269661 24201 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632440055392-2777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632440055392-2777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44967
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632440055392-2777-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632440055392-2777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:58:22.269860 24201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:58:22.270032 24201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:58:22.274863 24206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:22.274863 24209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:22.274885 24201 server_base.cc:1034] running on GCE node
W20260928 21:58:22.275573 24207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:22.275853 24201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:58:22.276196 24201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:58:22.277361 24201 hybrid_clock.cc:648] HybridClock initialized: now 1790632702277343 us; error 33 us; skew 500 ppm
Sep 28 21:58:22 dist-test-slave-k6lz krb5kdc[24188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632702, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:58:22.280056 24201 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 21:58:22.280598 24201 webserver.cc:460] Webserver started at http://127.0.0.1:41693/ using document root <none> and password file <none>
I20260928 21:58:22.280803 24201 fs_manager.cc:359] Metadata directory not provided
I20260928 21:58:22.280851 24201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:58:22.280987 24201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:58:22.282028 24201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632440055392-2777-0/minicluster-data/master-0/data/instance:
uuid: "8649211fcf884ab6868593020409cbce"
format_stamp: "Formatted at 2026-09-28 21:58:22 on dist-test-slave-k6lz"
server_key: "bc877435ff36795d26d38ff376fec81c"
server_key_iv: "0a7656e946eca771fb71b1dd25e2365a"
server_key_version: "encryptionkey@0"
I20260928 21:58:22.282349 24201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632440055392-2777-0/minicluster-data/master-0/wal/instance:
uuid: "8649211fcf884ab6868593020409cbce"
format_stamp: "Formatted at 2026-09-28 21:58:22 on dist-test-slave-k6lz"
server_key: "bc877435ff36795d26d38ff376fec81c"
server_key_iv: "0a7656e946eca771fb71b1dd25e2365a"
server_key_version: "encryptionkey@0"
I20260928 21:58:22.283648 24201 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:58:22.284389 24216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:22.284621 24201 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:58:22.284701 24201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632440055392-2777-0/minicluster-data/master-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632440055392-2777-0/minicluster-data/master-0/wal
uuid: "8649211fcf884ab6868593020409cbce"
format_stamp: "Formatted at 2026-09-28 21:58:22 on dist-test-slave-k6lz"
server_key: "bc877435ff36795d26d38ff376fec81c"
server_key_iv: "0a7656e946eca771fb71b1dd25e2365a"
server_key_version: "encryptionkey@0"
I20260928 21:58:22.284761 24201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632440055392-2777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632440055392-2777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632440055392-2777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:58:22.287982 24201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:58:22.289955 24201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:58:22.290078 24201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:58:22.293979 24201 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44967
I20260928 21:58:22.294039 24268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44967 every 8 connection(s)
I20260928 21:58:22.294337 24201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
I20260928 21:58:22.295470 24269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:58:22.297767 24269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8649211fcf884ab6868593020409cbce: Bootstrap starting.
I20260928 21:58:22.298317 24269 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8649211fcf884ab6868593020409cbce: Neither blocks nor log segments found. Creating new log.
I20260928 21:58:22.298615 24269 log.cc:824] T 00000000000000000000000000000000 P 8649211fcf884ab6868593020409cbce: Log is configured to *not* fsync() on all Append() calls
I20260928 21:58:22.299288 24269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8649211fcf884ab6868593020409cbce: No bootstrap required, opened a new log
I20260928 21:58:22.301009 24269 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8649211fcf884ab6868593020409cbce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8649211fcf884ab6868593020409cbce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44967 } }
I20260928 21:58:22.301153 24269 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8649211fcf884ab6868593020409cbce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:58:22.301175 24269 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8649211fcf884ab6868593020409cbce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8649211fcf884ab6868593020409cbce, State: Initialized, Role: FOLLOWER
I20260928 21:58:22.301302 24269 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8649211fcf884ab6868593020409cbce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8649211fcf884ab6868593020409cbce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44967 } }
I20260928 21:58:22.301368 24269 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8649211fcf884ab6868593020409cbce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:58:22.301396 24269 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8649211fcf884ab6868593020409cbce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:58:22.301455 24269 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8649211fcf884ab6868593020409cbce [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:58:22.301792 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 24201
I20260928 21:58:22.301924 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632440055392-2777-0/minicluster-data/master-0/wal/instance
I20260928 21:58:22.302091 2777 external_mini_cluster.cc:1428] Setting key 96ad5e1fd51c53770cf9a5d95cd4e236
I20260928 21:58:22.302156 24269 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8649211fcf884ab6868593020409cbce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8649211fcf884ab6868593020409cbce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44967 } }
I20260928 21:58:22.302259 24269 leader_election.cc:302] T 00000000000000000000000000000000 P 8649211fcf884ab6868593020409cbce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8649211fcf884ab6868593020409cbce; no voters:
I20260928 21:58:22.302430 24269 leader_election.cc:288] T 00000000000000000000000000000000 P 8649211fcf884ab6868593020409cbce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:58:22.302745 24269 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8649211fcf884ab6868593020409cbce [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:58:22.302448 24272 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8649211fcf884ab6868593020409cbce [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:58:22.304234 24272 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8649211fcf884ab6868593020409cbce [term 1 LEADER]: Becoming Leader. State: Replica: 8649211fcf884ab6868593020409cbce, State: Running, Role: LEADER
I20260928 21:58:22.304240 24275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:22.302578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.102:50439 (local address 127.0.0.1:44967)
0928 21:58:22.302726 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:22.302749 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:22.302794 (+ 45us) server_negotiation.cc:407] Connection header received
0928 21:58:22.302915 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:22.302921 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:22.302945 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:22.303059 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:22.303776 (+ 717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:22.303779 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:22.303780 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:58:22.303784 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:22.303824 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:58:22.303835 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:22.303888 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:22.303985 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 21:58:22.304044 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":58,"threads_started":1}
I20260928 21:58:22.304358 24272 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8649211fcf884ab6868593020409cbce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8649211fcf884ab6868593020409cbce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44967 } }
I20260928 21:58:22.304689 24272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8649211fcf884ab6868593020409cbce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8649211fcf884ab6868593020409cbce. Latest consensus state: current_term: 1 leader_uuid: "8649211fcf884ab6868593020409cbce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8649211fcf884ab6868593020409cbce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44967 } } }
I20260928 21:58:22.304757 24272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8649211fcf884ab6868593020409cbce [sys.catalog]: This master's current role is: LEADER
I20260928 21:58:22.304720 24286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8649211fcf884ab6868593020409cbce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8649211fcf884ab6868593020409cbce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8649211fcf884ab6868593020409cbce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44967 } } }
I20260928 21:58:22.304798 24286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8649211fcf884ab6868593020409cbce [sys.catalog]: This master's current role is: LEADER
I20260928 21:58:22.304955 24288 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:58:22.305269 2777 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:58:22.306697 24288 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:58:22.308055 24275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:22.306571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.102:33203 (local address 127.0.0.1:44967)
0928 21:58:22.306606 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:22.306608 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:22.306705 (+ 97us) server_negotiation.cc:407] Connection header received
0928 21:58:22.306793 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:22.306794 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:22.306799 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:22.306845 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:22.307838 (+ 993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:22.307839 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:22.307839 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:58:22.307843 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:22.307858 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:58:22.307862 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:22.307905 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:22.307962 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 21:58:22.307977 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 21:58:22.308367 24288 catalog_manager.cc:1347] Generated new cluster ID: 64146dbedaff49fb805470d7fb1575d7
I20260928 21:58:22.308420 24288 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:58:22.324221 24288 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:58:22.324426 24288 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:58:22.329043 24288 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8649211fcf884ab6868593020409cbce: Generated new TSK 0
I20260928 21:58:22.329219 24288 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:58:22.337028 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 24201
2026-09-28T21:58:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1241 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632440055392-2777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:58:22 dist-test-slave-k6lz krb5kdc[24300](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:58:22 dist-test-slave-k6lz krb5kdc[24300](info): set up 2 sockets
Sep 28 21:58:22 dist-test-slave-k6lz krb5kdc[24300](info): commencing operation
krb5kdc: starting...
W20260928 21:58:23.295486 2777 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Sep 28 21:58:23 dist-test-slave-k6lz krb5kdc[24300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:58:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:58:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab.
Entry for principal kudu/127.2.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab.
I20260928 21:58:23.418960 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632440055392-2777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632440055392-2777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632440055392-2777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:38713
--webserver_interface=127.2.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab
--principal=kudu/127.2.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:37517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.126:38713
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:58:23.498806 24313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:58:23.499017 24313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:58:23.499058 24313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:58:23.500276 24313 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:58:23.500327 24313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:58:23.500346 24313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:58:23.500362 24313 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:58:23.500377 24313 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:58:23.501878 24313 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:37517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632440055392-2777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632440055392-2777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.126:38713
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab
--principal=kudu/127.2.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:38713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632440055392-2777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:58:23.502149 24313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:58:23.502388 24313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:58:23.507293 24319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:23.507431 24318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:23.507486 24321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:23.507617 24313 server_base.cc:1034] running on GCE node
I20260928 21:58:23.507838 24313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:58:23.508139 24313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:58:23.509280 24313 hybrid_clock.cc:648] HybridClock initialized: now 1790632703509269 us; error 30 us; skew 500 ppm
Sep 28 21:58:23 dist-test-slave-k6lz krb5kdc[24300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632703, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:58:23.511696 24313 init.cc:377] Logged in from keytab as kudu/127.2.182.126@KRBTEST.COM (short username kudu)
I20260928 21:58:23.512156 24313 webserver.cc:460] Webserver started at http://127.2.182.126:45679/ using document root <none> and password file <none>
I20260928 21:58:23.512349 24313 fs_manager.cc:359] Metadata directory not provided
I20260928 21:58:23.512400 24313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:58:23.512522 24313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:58:23.513660 24313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632440055392-2777-0/minicluster-data/master-0/data/instance:
uuid: "8c8a5ef3ecc8445bb49d3de8ecbb7c3f"
format_stamp: "Formatted at 2026-09-28 21:58:23 on dist-test-slave-k6lz"
server_key: "cd78bb441ec7d6384545f5109ef1f1eb"
server_key_iv: "f2d46677de3fdab9e2b86d577db43e60"
server_key_version: "encryptionkey@0"
I20260928 21:58:23.513976 24313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632440055392-2777-0/minicluster-data/master-0/wal/instance:
uuid: "8c8a5ef3ecc8445bb49d3de8ecbb7c3f"
format_stamp: "Formatted at 2026-09-28 21:58:23 on dist-test-slave-k6lz"
server_key: "cd78bb441ec7d6384545f5109ef1f1eb"
server_key_iv: "f2d46677de3fdab9e2b86d577db43e60"
server_key_version: "encryptionkey@0"
I20260928 21:58:23.515304 24313 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 21:58:23.516081 24328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:23.516260 24313 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:58:23.516347 24313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632440055392-2777-0/minicluster-data/master-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632440055392-2777-0/minicluster-data/master-0/wal
uuid: "8c8a5ef3ecc8445bb49d3de8ecbb7c3f"
format_stamp: "Formatted at 2026-09-28 21:58:23 on dist-test-slave-k6lz"
server_key: "cd78bb441ec7d6384545f5109ef1f1eb"
server_key_iv: "f2d46677de3fdab9e2b86d577db43e60"
server_key_version: "encryptionkey@0"
I20260928 21:58:23.516408 24313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632440055392-2777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632440055392-2777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632440055392-2777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:58:23.524258 24313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:58:23.526260 24313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:58:23.526391 24313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:58:23.530119 24313 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.126:38713
I20260928 21:58:23.530155 24380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.126:38713 every 8 connection(s)
I20260928 21:58:23.530474 24313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
I20260928 21:58:23.531303 24381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:58:23.533596 24381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c8a5ef3ecc8445bb49d3de8ecbb7c3f: Bootstrap starting.
I20260928 21:58:23.533641 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 24313
I20260928 21:58:23.533751 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632440055392-2777-0/minicluster-data/master-0/wal/instance
I20260928 21:58:23.533919 2777 external_mini_cluster.cc:1428] Setting key e752916e34edfc126f6fdf3ab4dbdbc1
I20260928 21:58:23.534235 24381 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c8a5ef3ecc8445bb49d3de8ecbb7c3f: Neither blocks nor log segments found. Creating new log.
I20260928 21:58:23.534763 24381 log.cc:824] T 00000000000000000000000000000000 P 8c8a5ef3ecc8445bb49d3de8ecbb7c3f: Log is configured to *not* fsync() on all Append() calls
I20260928 21:58:23.535521 24381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c8a5ef3ecc8445bb49d3de8ecbb7c3f: No bootstrap required, opened a new log
I20260928 21:58:23.537822 24381 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c8a5ef3ecc8445bb49d3de8ecbb7c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c8a5ef3ecc8445bb49d3de8ecbb7c3f" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 38713 } }
I20260928 21:58:23.537955 24381 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c8a5ef3ecc8445bb49d3de8ecbb7c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:58:23.537976 24381 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c8a5ef3ecc8445bb49d3de8ecbb7c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c8a5ef3ecc8445bb49d3de8ecbb7c3f, State: Initialized, Role: FOLLOWER
I20260928 21:58:23.538076 24381 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c8a5ef3ecc8445bb49d3de8ecbb7c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c8a5ef3ecc8445bb49d3de8ecbb7c3f" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 38713 } }
I20260928 21:58:23.538151 24381 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c8a5ef3ecc8445bb49d3de8ecbb7c3f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:58:23.538173 24381 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c8a5ef3ecc8445bb49d3de8ecbb7c3f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:58:23.538216 24381 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c8a5ef3ecc8445bb49d3de8ecbb7c3f [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:58:23.538873 24381 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c8a5ef3ecc8445bb49d3de8ecbb7c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c8a5ef3ecc8445bb49d3de8ecbb7c3f" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 38713 } }
I20260928 21:58:23.538980 24381 leader_election.cc:302] T 00000000000000000000000000000000 P 8c8a5ef3ecc8445bb49d3de8ecbb7c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c8a5ef3ecc8445bb49d3de8ecbb7c3f; no voters:
I20260928 21:58:23.539153 24381 leader_election.cc:288] T 00000000000000000000000000000000 P 8c8a5ef3ecc8445bb49d3de8ecbb7c3f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:58:23.539202 24386 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c8a5ef3ecc8445bb49d3de8ecbb7c3f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:58:23.539371 24386 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c8a5ef3ecc8445bb49d3de8ecbb7c3f [term 1 LEADER]: Becoming Leader. State: Replica: 8c8a5ef3ecc8445bb49d3de8ecbb7c3f, State: Running, Role: LEADER
I20260928 21:58:23.539445 24381 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c8a5ef3ecc8445bb49d3de8ecbb7c3f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:58:23.539491 24386 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c8a5ef3ecc8445bb49d3de8ecbb7c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c8a5ef3ecc8445bb49d3de8ecbb7c3f" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 38713 } }
I20260928 21:58:23.539939 24388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c8a5ef3ecc8445bb49d3de8ecbb7c3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c8a5ef3ecc8445bb49d3de8ecbb7c3f. Latest consensus state: current_term: 1 leader_uuid: "8c8a5ef3ecc8445bb49d3de8ecbb7c3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c8a5ef3ecc8445bb49d3de8ecbb7c3f" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 38713 } } }
I20260928 21:58:23.540150 24388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c8a5ef3ecc8445bb49d3de8ecbb7c3f [sys.catalog]: This master's current role is: LEADER
I20260928 21:58:23.540091 24387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c8a5ef3ecc8445bb49d3de8ecbb7c3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c8a5ef3ecc8445bb49d3de8ecbb7c3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c8a5ef3ecc8445bb49d3de8ecbb7c3f" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 38713 } } }
I20260928 21:58:23.540369 24387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c8a5ef3ecc8445bb49d3de8ecbb7c3f [sys.catalog]: This master's current role is: LEADER
W20260928 21:58:23.541191 24402 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8c8a5ef3ecc8445bb49d3de8ecbb7c3f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:58:23.541281 24402 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 28 21:58:23 dist-test-slave-k6lz krb5kdc[24300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:58:23.541399 24395 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:58:23.541603 24395 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:58:23.543411 24395 catalog_manager.cc:1347] Generated new cluster ID: 36db2ca54acc4f8592307109f5436555
I20260928 21:58:23.543454 24395 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:58:23.545617 24384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:23.534429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59231 (local address 127.2.182.126:38713)
0928 21:58:23.534590 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:23.534608 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:23.534645 (+ 37us) server_negotiation.cc:407] Connection header received
0928 21:58:23.535168 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:23.535177 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:23.535239 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:23.535418 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:23.536117 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:23.537482 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:23.538353 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:23.538941 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:23.539047 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:23.541959 (+ 2912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:23.541963 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:23.541964 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:23.541984 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:23.543922 (+ 1938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:23.544386 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:23.544389 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:23.544389 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:23.544491 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:23.544724 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:23.544724 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:23.544725 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:23.545001 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:23.545146 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:23.545291 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 21:58:23.545384 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":64,"threads_started":1}
I20260928 21:58:23.546582 2777 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:58:23.555215 24384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:23.548208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48367 (local address 127.2.182.126:38713)
0928 21:58:23.548237 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:23.548239 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:23.548262 (+ 23us) server_negotiation.cc:407] Connection header received
0928 21:58:23.548280 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:23.548280 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:23.548289 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:23.548332 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:23.548944 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:23.549667 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:23.550650 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:23.550960 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:23.551009 (+ 49us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:23.552512 (+ 1503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:23.552514 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:23.552514 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:23.552521 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:23.553961 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:23.554359 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:23.554360 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:23.554360 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:23.554425 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:23.554620 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:23.554621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:23.554621 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:23.554738 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:23.554843 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:23.554961 (+ 118us) server_negotiation.cc:299] Negotiation successful
0928 21:58:23.555018 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13}
I20260928 21:58:23.563166 24395 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:58:23.563735 24395 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:58:23.569934 24395 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c8a5ef3ecc8445bb49d3de8ecbb7c3f: Generated new TSK 0
I20260928 21:58:23.570151 24395 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:58:23.572338 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 24313
W20260928 21:58:23.576510 24298 connection.cc:535] client connection to 127.2.182.126:38713 recv error: Network error: failed to read from TLS socket (remote: 127.2.182.126:38713): Connection reset by peer (error 104)
2026-09-28T21:58:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1235 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632440055392-2777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:58:23 dist-test-slave-k6lz krb5kdc[24412](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:58:23 dist-test-slave-k6lz krb5kdc[24412](info): set up 2 sockets
Sep 28 21:58:23 dist-test-slave-k6lz krb5kdc[24412](info): commencing operation
krb5kdc: starting...
W20260928 21:58:24.527213 2777 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Sep 28 21:58:24 dist-test-slave-k6lz krb5kdc[24412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:58:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:58:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab.
Entry for principal kudu/127.2.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab.
I20260928 21:58:24.645551 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632440055392-2777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632440055392-2777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632440055392-2777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:45491
--webserver_interface=127.2.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab
--principal=kudu/127.2.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:44099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.126:45491
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:58:24.725951 24425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:58:24.726166 24425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:58:24.726187 24425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:58:24.727291 24425 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:58:24.727324 24425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:58:24.727370 24425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:58:24.727380 24425 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:58:24.727397 24425 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:58:24.728844 24425 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:44099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632440055392-2777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632440055392-2777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.126:45491
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab
--principal=kudu/127.2.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:45491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632440055392-2777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:58:24.729039 24425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:58:24.729214 24425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:58:24.734082 24431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:24.734123 24425 server_base.cc:1034] running on GCE node
W20260928 21:58:24.734090 24430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:24.734102 24433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:24.734586 24425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:58:24.734862 24425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:58:24.736016 24425 hybrid_clock.cc:648] HybridClock initialized: now 1790632704736000 us; error 30 us; skew 500 ppm
Sep 28 21:58:24 dist-test-slave-k6lz krb5kdc[24412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632704, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:58:24.738397 24425 init.cc:377] Logged in from keytab as kudu/127.2.182.126@KRBTEST.COM (short username kudu)
I20260928 21:58:24.738898 24425 webserver.cc:460] Webserver started at http://127.2.182.126:43563/ using document root <none> and password file <none>
I20260928 21:58:24.739133 24425 fs_manager.cc:359] Metadata directory not provided
I20260928 21:58:24.739194 24425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:58:24.739312 24425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:58:24.740397 24425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632440055392-2777-0/minicluster-data/master-0/data/instance:
uuid: "2e464165b6c84e9688f4d3536eb1f574"
format_stamp: "Formatted at 2026-09-28 21:58:24 on dist-test-slave-k6lz"
server_key: "25b8b5866c0596dd86899c38a98e8291"
server_key_iv: "c794b153a286b49fc4ab4935ac20fef1"
server_key_version: "encryptionkey@0"
I20260928 21:58:24.740710 24425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632440055392-2777-0/minicluster-data/master-0/wal/instance:
uuid: "2e464165b6c84e9688f4d3536eb1f574"
format_stamp: "Formatted at 2026-09-28 21:58:24 on dist-test-slave-k6lz"
server_key: "25b8b5866c0596dd86899c38a98e8291"
server_key_iv: "c794b153a286b49fc4ab4935ac20fef1"
server_key_version: "encryptionkey@0"
I20260928 21:58:24.741905 24425 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 21:58:24.742681 24440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:24.742841 24425 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:58:24.742931 24425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632440055392-2777-0/minicluster-data/master-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632440055392-2777-0/minicluster-data/master-0/wal
uuid: "2e464165b6c84e9688f4d3536eb1f574"
format_stamp: "Formatted at 2026-09-28 21:58:24 on dist-test-slave-k6lz"
server_key: "25b8b5866c0596dd86899c38a98e8291"
server_key_iv: "c794b153a286b49fc4ab4935ac20fef1"
server_key_version: "encryptionkey@0"
I20260928 21:58:24.742992 24425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632440055392-2777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632440055392-2777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632440055392-2777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:58:24.756511 24425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:58:24.758492 24425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:58:24.758623 24425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:58:24.762419 24425 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.126:45491
I20260928 21:58:24.762518 24492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.126:45491 every 8 connection(s)
I20260928 21:58:24.762806 24425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
I20260928 21:58:24.763773 24493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:58:24.765970 24493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e464165b6c84e9688f4d3536eb1f574: Bootstrap starting.
I20260928 21:58:24.766521 24493 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e464165b6c84e9688f4d3536eb1f574: Neither blocks nor log segments found. Creating new log.
I20260928 21:58:24.766767 24493 log.cc:824] T 00000000000000000000000000000000 P 2e464165b6c84e9688f4d3536eb1f574: Log is configured to *not* fsync() on all Append() calls
I20260928 21:58:24.767496 24493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e464165b6c84e9688f4d3536eb1f574: No bootstrap required, opened a new log
I20260928 21:58:24.769081 24493 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e464165b6c84e9688f4d3536eb1f574 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e464165b6c84e9688f4d3536eb1f574" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 45491 } }
I20260928 21:58:24.769218 24493 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e464165b6c84e9688f4d3536eb1f574 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:58:24.769239 24493 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e464165b6c84e9688f4d3536eb1f574 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e464165b6c84e9688f4d3536eb1f574, State: Initialized, Role: FOLLOWER
I20260928 21:58:24.769366 24493 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e464165b6c84e9688f4d3536eb1f574 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e464165b6c84e9688f4d3536eb1f574" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 45491 } }
I20260928 21:58:24.769433 24493 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e464165b6c84e9688f4d3536eb1f574 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:58:24.769460 24493 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e464165b6c84e9688f4d3536eb1f574 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:58:24.769492 24493 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e464165b6c84e9688f4d3536eb1f574 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:58:24.770298 24493 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e464165b6c84e9688f4d3536eb1f574 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e464165b6c84e9688f4d3536eb1f574" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 45491 } }
I20260928 21:58:24.770401 24493 leader_election.cc:302] T 00000000000000000000000000000000 P 2e464165b6c84e9688f4d3536eb1f574 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e464165b6c84e9688f4d3536eb1f574; no voters:
I20260928 21:58:24.770524 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 24425
I20260928 21:58:24.770556 24493 leader_election.cc:288] T 00000000000000000000000000000000 P 2e464165b6c84e9688f4d3536eb1f574 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:58:24.770640 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632440055392-2777-0/minicluster-data/master-0/wal/instance
I20260928 21:58:24.770733 24496 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e464165b6c84e9688f4d3536eb1f574 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:58:24.770829 2777 external_mini_cluster.cc:1428] Setting key 0f929fac462fbcf7aca3b61283a4a8bb
I20260928 21:58:24.770828 24493 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e464165b6c84e9688f4d3536eb1f574 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:58:24.770941 24496 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e464165b6c84e9688f4d3536eb1f574 [term 1 LEADER]: Becoming Leader. State: Replica: 2e464165b6c84e9688f4d3536eb1f574, State: Running, Role: LEADER
I20260928 21:58:24.771064 24496 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e464165b6c84e9688f4d3536eb1f574 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e464165b6c84e9688f4d3536eb1f574" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 45491 } }
I20260928 21:58:24.771478 24497 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e464165b6c84e9688f4d3536eb1f574 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e464165b6c84e9688f4d3536eb1f574" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e464165b6c84e9688f4d3536eb1f574" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 45491 } } }
I20260928 21:58:24.771508 24498 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e464165b6c84e9688f4d3536eb1f574 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e464165b6c84e9688f4d3536eb1f574. Latest consensus state: current_term: 1 leader_uuid: "2e464165b6c84e9688f4d3536eb1f574" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e464165b6c84e9688f4d3536eb1f574" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 45491 } } }
I20260928 21:58:24.771566 24497 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e464165b6c84e9688f4d3536eb1f574 [sys.catalog]: This master's current role is: LEADER
I20260928 21:58:24.771569 24498 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e464165b6c84e9688f4d3536eb1f574 [sys.catalog]: This master's current role is: LEADER
I20260928 21:58:24.771871 24507 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:58:24.772570 24507 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:58:24.774295 24507 catalog_manager.cc:1347] Generated new cluster ID: a711fd044930422593dd41b8135f2e2b
I20260928 21:58:24.774353 24507 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:58:24.777113 24504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:24.771348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42761 (local address 127.2.182.126:45491)
0928 21:58:24.771741 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:24.771751 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:24.771793 (+ 42us) server_negotiation.cc:407] Connection header received
0928 21:58:24.771931 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:24.771940 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:24.772007 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:24.772367 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:24.772464 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:24.773847 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:24.774901 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:24.775464 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:24.775751 (+ 287us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:24.776493 (+ 742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:24.776497 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:24.776497 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:58:24.776665 (+ 168us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:24.776696 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:58:24.776708 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:24.776777 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:24.776868 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 21:58:24.776909 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":79,"threads_started":1}
I20260928 21:58:24.778079 2777 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:58:24.786628 24504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:24.783751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56053 (local address 127.2.182.126:45491)
0928 21:58:24.783795 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:24.783798 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:24.783955 (+ 157us) server_negotiation.cc:407] Connection header received
0928 21:58:24.784016 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:24.784017 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:24.784029 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:24.784076 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:24.784218 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:24.784708 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:24.785434 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:24.785699 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:24.785759 (+ 60us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:24.786387 (+ 628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:24.786389 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:24.786389 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:58:24.786394 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:24.786408 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:58:24.786412 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:24.786458 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:24.786524 (+ 66us) server_negotiation.cc:299] Negotiation successful
0928 21:58:24.786535 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 21:58:24.790776 24507 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:58:24.791504 24507 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:58:24.800987 24507 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e464165b6c84e9688f4d3536eb1f574: Generated new TSK 0
I20260928 21:58:24.801208 24507 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:58:24.803573 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 24425
W20260928 21:58:24.807089 24410 connection.cc:535] client connection to 127.2.182.126:45491 recv error: Network error: failed to read from TLS socket (remote: 127.2.182.126:45491): Connection reset by peer (error 104)
2026-09-28T21:58:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1230 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632440055392-2777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:58:24 dist-test-slave-k6lz krb5kdc[24524](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:58:24 dist-test-slave-k6lz krb5kdc[24524](info): set up 2 sockets
Sep 28 21:58:24 dist-test-slave-k6lz krb5kdc[24524](info): commencing operation
krb5kdc: starting...
W20260928 21:58:25.755190 2777 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Sep 28 21:58:25 dist-test-slave-k6lz krb5kdc[24524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632705, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:58:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:58:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab.
Entry for principal kudu/127.2.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab.
I20260928 21:58:25.875260 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632440055392-2777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632440055392-2777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632440055392-2777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:44943
--webserver_interface=127.2.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab
--principal=kudu/127.2.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:33685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.126:44943
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:58:25.954051 24537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:58:25.954272 24537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:58:25.954303 24537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:58:25.955557 24537 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:58:25.955607 24537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:58:25.955626 24537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:58:25.955642 24537 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:58:25.955657 24537 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:58:25.957150 24537 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:33685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632440055392-2777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632440055392-2777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.126:44943
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab
--principal=kudu/127.2.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:44943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24537
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632440055392-2777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:58:25.957406 24537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:58:25.957631 24537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:58:25.962445 24543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:25.962450 24545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:25.962440 24542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:25.962836 24537 server_base.cc:1034] running on GCE node
I20260928 21:58:25.963079 24537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:58:25.963395 24537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:58:25.964620 24537 hybrid_clock.cc:648] HybridClock initialized: now 1790632705964599 us; error 39 us; skew 500 ppm
Sep 28 21:58:25 dist-test-slave-k6lz krb5kdc[24524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632705, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:58:25.973701 24537 init.cc:377] Logged in from keytab as kudu/127.2.182.126@KRBTEST.COM (short username kudu)
I20260928 21:58:25.974177 24537 webserver.cc:460] Webserver started at http://127.2.182.126:37979/ using document root <none> and password file <none>
I20260928 21:58:25.974375 24537 fs_manager.cc:359] Metadata directory not provided
I20260928 21:58:25.974433 24537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:58:25.974560 24537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:58:25.975637 24537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632440055392-2777-0/minicluster-data/master-0/data/instance:
uuid: "da63a2292cd94175b82cfef5086e53e9"
format_stamp: "Formatted at 2026-09-28 21:58:25 on dist-test-slave-k6lz"
server_key: "520bd6f07c180729bbf3f8d739042869"
server_key_iv: "283e0c15463ee780203d2e355ed38d1f"
server_key_version: "encryptionkey@0"
I20260928 21:58:25.975958 24537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632440055392-2777-0/minicluster-data/master-0/wal/instance:
uuid: "da63a2292cd94175b82cfef5086e53e9"
format_stamp: "Formatted at 2026-09-28 21:58:25 on dist-test-slave-k6lz"
server_key: "520bd6f07c180729bbf3f8d739042869"
server_key_iv: "283e0c15463ee780203d2e355ed38d1f"
server_key_version: "encryptionkey@0"
I20260928 21:58:25.977216 24537 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:58:25.978008 24552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:25.978176 24537 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:58:25.978267 24537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632440055392-2777-0/minicluster-data/master-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632440055392-2777-0/minicluster-data/master-0/wal
uuid: "da63a2292cd94175b82cfef5086e53e9"
format_stamp: "Formatted at 2026-09-28 21:58:25 on dist-test-slave-k6lz"
server_key: "520bd6f07c180729bbf3f8d739042869"
server_key_iv: "283e0c15463ee780203d2e355ed38d1f"
server_key_version: "encryptionkey@0"
I20260928 21:58:25.978328 24537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632440055392-2777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632440055392-2777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632440055392-2777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:58:25.981693 24537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:58:25.983817 24537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:58:25.983956 24537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:58:25.987488 24537 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.126:44943
I20260928 21:58:25.987526 24604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.126:44943 every 8 connection(s)
I20260928 21:58:25.987826 24537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
I20260928 21:58:25.988772 24605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:58:25.989984 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 24537
I20260928 21:58:25.990103 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632440055392-2777-0/minicluster-data/master-0/wal/instance
I20260928 21:58:25.990296 2777 external_mini_cluster.cc:1428] Setting key 7821fcda56322d0391d9d2fd132e0243
I20260928 21:58:25.991389 24605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da63a2292cd94175b82cfef5086e53e9: Bootstrap starting.
I20260928 21:58:25.992089 24605 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da63a2292cd94175b82cfef5086e53e9: Neither blocks nor log segments found. Creating new log.
I20260928 21:58:25.992377 24605 log.cc:824] T 00000000000000000000000000000000 P da63a2292cd94175b82cfef5086e53e9: Log is configured to *not* fsync() on all Append() calls
I20260928 21:58:25.992916 24608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:25.990895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40209 (local address 127.2.182.126:44943)
0928 21:58:25.991083 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:25.991092 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:25.991126 (+ 34us) server_negotiation.cc:407] Connection header received
0928 21:58:25.991732 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:25.991743 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:25.991779 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:25.991942 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:25.992539 (+ 597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:25.992543 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:25.992543 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:58:25.992547 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:25.992576 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:58:25.992586 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:25.992646 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:25.992708 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 21:58:25.992740 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":84,"threads_started":1}
I20260928 21:58:25.993007 24605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da63a2292cd94175b82cfef5086e53e9: No bootstrap required, opened a new log
I20260928 21:58:25.993824 2777 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:58:25.994982 24605 raft_consensus.cc:348] T 00000000000000000000000000000000 P da63a2292cd94175b82cfef5086e53e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da63a2292cd94175b82cfef5086e53e9" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 44943 } }
I20260928 21:58:25.995119 24605 raft_consensus.cc:374] T 00000000000000000000000000000000 P da63a2292cd94175b82cfef5086e53e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:58:25.995150 24605 raft_consensus.cc:729] T 00000000000000000000000000000000 P da63a2292cd94175b82cfef5086e53e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da63a2292cd94175b82cfef5086e53e9, State: Initialized, Role: FOLLOWER
I20260928 21:58:25.995258 24605 consensus_queue.cc:260] T 00000000000000000000000000000000 P da63a2292cd94175b82cfef5086e53e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da63a2292cd94175b82cfef5086e53e9" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 44943 } }
I20260928 21:58:25.995398 24605 raft_consensus.cc:388] T 00000000000000000000000000000000 P da63a2292cd94175b82cfef5086e53e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:58:25.995447 24605 raft_consensus.cc:482] T 00000000000000000000000000000000 P da63a2292cd94175b82cfef5086e53e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:58:25.995532 24605 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da63a2292cd94175b82cfef5086e53e9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:58:25.996088 24605 raft_consensus.cc:504] T 00000000000000000000000000000000 P da63a2292cd94175b82cfef5086e53e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da63a2292cd94175b82cfef5086e53e9" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 44943 } }
I20260928 21:58:25.996187 24608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:25.995172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33275 (local address 127.2.182.126:44943)
0928 21:58:25.995213 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:25.995215 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:25.995309 (+ 94us) server_negotiation.cc:407] Connection header received
0928 21:58:25.995391 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:25.995392 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:25.995397 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:25.995454 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:25.995987 (+ 533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:25.995988 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:25.995988 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:58:25.995993 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:25.996015 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:58:25.996019 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:25.996084 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:25.996125 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 21:58:25.996137 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 21:58:25.996227 24605 leader_election.cc:302] T 00000000000000000000000000000000 P da63a2292cd94175b82cfef5086e53e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da63a2292cd94175b82cfef5086e53e9; no voters:
I20260928 21:58:25.996358 24605 leader_election.cc:288] T 00000000000000000000000000000000 P da63a2292cd94175b82cfef5086e53e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:58:25.996392 24615 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da63a2292cd94175b82cfef5086e53e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:58:25.996508 24615 raft_consensus.cc:686] T 00000000000000000000000000000000 P da63a2292cd94175b82cfef5086e53e9 [term 1 LEADER]: Becoming Leader. State: Replica: da63a2292cd94175b82cfef5086e53e9, State: Running, Role: LEADER
I20260928 21:58:25.996627 24605 sys_catalog.cc:564] T 00000000000000000000000000000000 P da63a2292cd94175b82cfef5086e53e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:58:25.996642 24615 consensus_queue.cc:237] T 00000000000000000000000000000000 P da63a2292cd94175b82cfef5086e53e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da63a2292cd94175b82cfef5086e53e9" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 44943 } }
I20260928 21:58:25.996956 24615 sys_catalog.cc:455] T 00000000000000000000000000000000 P da63a2292cd94175b82cfef5086e53e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader da63a2292cd94175b82cfef5086e53e9. Latest consensus state: current_term: 1 leader_uuid: "da63a2292cd94175b82cfef5086e53e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da63a2292cd94175b82cfef5086e53e9" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 44943 } } }
I20260928 21:58:25.997012 24616 sys_catalog.cc:455] T 00000000000000000000000000000000 P da63a2292cd94175b82cfef5086e53e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da63a2292cd94175b82cfef5086e53e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da63a2292cd94175b82cfef5086e53e9" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 44943 } } }
I20260928 21:58:25.997089 24615 sys_catalog.cc:458] T 00000000000000000000000000000000 P da63a2292cd94175b82cfef5086e53e9 [sys.catalog]: This master's current role is: LEADER
I20260928 21:58:25.997189 24616 sys_catalog.cc:458] T 00000000000000000000000000000000 P da63a2292cd94175b82cfef5086e53e9 [sys.catalog]: This master's current role is: LEADER
I20260928 21:58:25.997339 24622 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:58:25.997855 24622 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:58:25.999442 24622 catalog_manager.cc:1347] Generated new cluster ID: f8c5eb970a954f9a81ab91fd1f41edd4
I20260928 21:58:25.999508 24622 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:58:26.018205 24622 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:58:26.018396 24622 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:58:26.022508 24622 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da63a2292cd94175b82cfef5086e53e9: Generated new TSK 0
I20260928 21:58:26.022717 24622 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:58:26.025161 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 24537
2026-09-28T21:58:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1224 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632440055392-2777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:58:26 dist-test-slave-k6lz krb5kdc[24636](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:58:26 dist-test-slave-k6lz krb5kdc[24636](info): set up 2 sockets
Sep 28 21:58:26 dist-test-slave-k6lz krb5kdc[24636](info): commencing operation
krb5kdc: starting...
W20260928 21:58:26.987183 2777 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Sep 28 21:58:27 dist-test-slave-k6lz krb5kdc[24636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:58:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:58:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab.
Entry for principal kudu/127.2.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab.
I20260928 21:58:27.102856 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632440055392-2777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632440055392-2777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632440055392-2777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:40887
--webserver_interface=127.2.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab
--principal=kudu/127.2.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:39085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.126:40887
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:58:27.181578 24649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:58:27.181784 24649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:58:27.181815 24649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:58:27.182950 24649 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:58:27.182991 24649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:58:27.183008 24649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:58:27.183027 24649 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:58:27.183043 24649 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:58:27.184514 24649 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:39085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632440055392-2777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632440055392-2777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.126:40887
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab
--principal=kudu/127.2.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:40887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24649
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632440055392-2777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:58:27.184746 24649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:58:27.184952 24649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:58:27.189658 24657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:27.189639 24654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:27.189769 24655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:27.189790 24649 server_base.cc:1034] running on GCE node
I20260928 21:58:27.190112 24649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:58:27.190382 24649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:58:27.191512 24649 hybrid_clock.cc:648] HybridClock initialized: now 1790632707191495 us; error 34 us; skew 500 ppm
Sep 28 21:58:27 dist-test-slave-k6lz krb5kdc[24636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632707, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:58:27.193886 24649 init.cc:377] Logged in from keytab as kudu/127.2.182.126@KRBTEST.COM (short username kudu)
I20260928 21:58:27.194315 24649 webserver.cc:460] Webserver started at http://127.2.182.126:43523/ using document root <none> and password file <none>
I20260928 21:58:27.194531 24649 fs_manager.cc:359] Metadata directory not provided
I20260928 21:58:27.194579 24649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:58:27.194732 24649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:58:27.195802 24649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632440055392-2777-0/minicluster-data/master-0/data/instance:
uuid: "7799918d3f5c4c1cb73fdd4b61ba1abe"
format_stamp: "Formatted at 2026-09-28 21:58:27 on dist-test-slave-k6lz"
server_key: "0de2545e921c39dae69dfeac2b0bc5da"
server_key_iv: "535a638642638b1b16f5f66ba8c47467"
server_key_version: "encryptionkey@0"
I20260928 21:58:27.196113 24649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632440055392-2777-0/minicluster-data/master-0/wal/instance:
uuid: "7799918d3f5c4c1cb73fdd4b61ba1abe"
format_stamp: "Formatted at 2026-09-28 21:58:27 on dist-test-slave-k6lz"
server_key: "0de2545e921c39dae69dfeac2b0bc5da"
server_key_iv: "535a638642638b1b16f5f66ba8c47467"
server_key_version: "encryptionkey@0"
I20260928 21:58:27.197338 24649 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:58:27.198128 24664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:27.198309 24649 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:58:27.198390 24649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632440055392-2777-0/minicluster-data/master-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632440055392-2777-0/minicluster-data/master-0/wal
uuid: "7799918d3f5c4c1cb73fdd4b61ba1abe"
format_stamp: "Formatted at 2026-09-28 21:58:27 on dist-test-slave-k6lz"
server_key: "0de2545e921c39dae69dfeac2b0bc5da"
server_key_iv: "535a638642638b1b16f5f66ba8c47467"
server_key_version: "encryptionkey@0"
I20260928 21:58:27.198449 24649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632440055392-2777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632440055392-2777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632440055392-2777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:58:27.218122 24649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:58:27.220253 24649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:58:27.220388 24649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:58:27.224445 24649 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.126:40887
I20260928 21:58:27.224522 24716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.126:40887 every 8 connection(s)
I20260928 21:58:27.224817 24649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
I20260928 21:58:27.225605 24717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:58:27.227681 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 24649
I20260928 21:58:27.227785 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632440055392-2777-0/minicluster-data/master-0/wal/instance
I20260928 21:58:27.227969 2777 external_mini_cluster.cc:1428] Setting key 27c87e74b83613f0ccb7d4860121eff0
I20260928 21:58:27.228014 24717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7799918d3f5c4c1cb73fdd4b61ba1abe: Bootstrap starting.
I20260928 21:58:27.228650 24717 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7799918d3f5c4c1cb73fdd4b61ba1abe: Neither blocks nor log segments found. Creating new log.
I20260928 21:58:27.228920 24717 log.cc:824] T 00000000000000000000000000000000 P 7799918d3f5c4c1cb73fdd4b61ba1abe: Log is configured to *not* fsync() on all Append() calls
I20260928 21:58:27.229584 24717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7799918d3f5c4c1cb73fdd4b61ba1abe: No bootstrap required, opened a new log
I20260928 21:58:27.231616 24717 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7799918d3f5c4c1cb73fdd4b61ba1abe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7799918d3f5c4c1cb73fdd4b61ba1abe" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 40887 } }
I20260928 21:58:27.231781 24717 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7799918d3f5c4c1cb73fdd4b61ba1abe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:58:27.231825 24717 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7799918d3f5c4c1cb73fdd4b61ba1abe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7799918d3f5c4c1cb73fdd4b61ba1abe, State: Initialized, Role: FOLLOWER
I20260928 21:58:27.231943 24717 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7799918d3f5c4c1cb73fdd4b61ba1abe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7799918d3f5c4c1cb73fdd4b61ba1abe" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 40887 } }
I20260928 21:58:27.232036 24717 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7799918d3f5c4c1cb73fdd4b61ba1abe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:58:27.232065 24717 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7799918d3f5c4c1cb73fdd4b61ba1abe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:58:27.232110 24717 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7799918d3f5c4c1cb73fdd4b61ba1abe [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:58:27.232674 24717 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7799918d3f5c4c1cb73fdd4b61ba1abe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7799918d3f5c4c1cb73fdd4b61ba1abe" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 40887 } }
I20260928 21:58:27.232780 24717 leader_election.cc:302] T 00000000000000000000000000000000 P 7799918d3f5c4c1cb73fdd4b61ba1abe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7799918d3f5c4c1cb73fdd4b61ba1abe; no voters:
I20260928 21:58:27.232941 24717 leader_election.cc:288] T 00000000000000000000000000000000 P 7799918d3f5c4c1cb73fdd4b61ba1abe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:58:27.233088 24723 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7799918d3f5c4c1cb73fdd4b61ba1abe [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:58:27.233182 24717 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7799918d3f5c4c1cb73fdd4b61ba1abe [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:58:27.233261 24723 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7799918d3f5c4c1cb73fdd4b61ba1abe [term 1 LEADER]: Becoming Leader. State: Replica: 7799918d3f5c4c1cb73fdd4b61ba1abe, State: Running, Role: LEADER
I20260928 21:58:27.233361 24723 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7799918d3f5c4c1cb73fdd4b61ba1abe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7799918d3f5c4c1cb73fdd4b61ba1abe" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 40887 } }
I20260928 21:58:27.233675 24722 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7799918d3f5c4c1cb73fdd4b61ba1abe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7799918d3f5c4c1cb73fdd4b61ba1abe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7799918d3f5c4c1cb73fdd4b61ba1abe" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 40887 } } }
I20260928 21:58:27.233673 24724 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7799918d3f5c4c1cb73fdd4b61ba1abe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7799918d3f5c4c1cb73fdd4b61ba1abe. Latest consensus state: current_term: 1 leader_uuid: "7799918d3f5c4c1cb73fdd4b61ba1abe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7799918d3f5c4c1cb73fdd4b61ba1abe" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 40887 } } }
I20260928 21:58:27.233767 24722 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7799918d3f5c4c1cb73fdd4b61ba1abe [sys.catalog]: This master's current role is: LEADER
I20260928 21:58:27.233770 24724 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7799918d3f5c4c1cb73fdd4b61ba1abe [sys.catalog]: This master's current role is: LEADER
I20260928 21:58:27.234424 24731 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 28 21:58:27 dist-test-slave-k6lz krb5kdc[24636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:58:27.234894 24731 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:58:27.236213 24731 catalog_manager.cc:1347] Generated new cluster ID: 50b1f895cd174a58a82cd8b7ad3ea62b
I20260928 21:58:27.236260 24731 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:58:27.242029 24720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:27.228488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42563 (local address 127.2.182.126:40887)
0928 21:58:27.228653 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:27.228660 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:27.228684 (+ 24us) server_negotiation.cc:407] Connection header received
0928 21:58:27.229155 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:27.229161 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:27.229209 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:27.229318 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:27.230032 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:27.230938 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:27.231983 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:27.232474 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:27.232561 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:27.235249 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:27.235252 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:27.235253 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:27.235264 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:27.240197 (+ 4933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:27.240567 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:27.240570 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:27.240571 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:27.240662 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:27.240848 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:27.240849 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:27.240849 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:27.241092 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:27.241693 (+ 601us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:27.241757 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 21:58:27.241876 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":61,"threads_started":1}
I20260928 21:58:27.242887 2777 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:58:27.249500 24731 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:58:27.250180 24731 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:58:27.251791 24720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:27.244229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40071 (local address 127.2.182.126:40887)
0928 21:58:27.244270 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:27.244272 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:27.244306 (+ 34us) server_negotiation.cc:407] Connection header received
0928 21:58:27.244363 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:27.244364 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:27.244373 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:27.244431 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:27.244907 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:27.245440 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:27.246114 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:27.246325 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:27.246377 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:27.247470 (+ 1093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:27.247472 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:27.247472 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:27.247479 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:27.248677 (+ 1198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:27.248971 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:27.248972 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:27.248972 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:27.249026 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:27.249224 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:27.249225 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:27.249225 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:27.249340 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:27.249428 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:27.251619 (+ 2191us) server_negotiation.cc:299] Negotiation successful
0928 21:58:27.251707 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 21:58:27.258662 24731 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7799918d3f5c4c1cb73fdd4b61ba1abe: Generated new TSK 0
I20260928 21:58:27.258850 24731 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:58:27.263543 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 24649
2026-09-28T21:58:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1236 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632440055392-2777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:58:27 dist-test-slave-k6lz krb5kdc[24748](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:58:27 dist-test-slave-k6lz krb5kdc[24748](info): set up 2 sockets
Sep 28 21:58:27 dist-test-slave-k6lz krb5kdc[24748](info): commencing operation
krb5kdc: starting...
W20260928 21:58:28.227170 2777 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Sep 28 21:58:28 dist-test-slave-k6lz krb5kdc[24748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632708, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:58:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:58:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab.
Entry for principal kudu/127.2.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab.
I20260928 21:58:28.346886 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632440055392-2777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632440055392-2777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632440055392-2777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:34515
--webserver_interface=127.2.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab
--principal=kudu/127.2.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:46295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.126:34515
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:58:28.428932 24761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:58:28.429316 24761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:58:28.429358 24761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:58:28.430826 24761 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:58:28.430883 24761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:58:28.430902 24761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:58:28.430919 24761 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:58:28.430935 24761 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:58:28.432662 24761 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:46295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632440055392-2777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632440055392-2777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.126:34515
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab
--principal=kudu/127.2.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:34515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632440055392-2777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:58:28.432933 24761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:58:28.433262 24761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:58:28.438606 24766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:28.438637 24767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:28.438649 24769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:28.438761 24761 server_base.cc:1034] running on GCE node
I20260928 21:58:28.439148 24761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:58:28.439481 24761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:58:28.440647 24761 hybrid_clock.cc:648] HybridClock initialized: now 1790632708440632 us; error 28 us; skew 500 ppm
Sep 28 21:58:28 dist-test-slave-k6lz krb5kdc[24748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632708, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:58:28.443158 24761 init.cc:377] Logged in from keytab as kudu/127.2.182.126@KRBTEST.COM (short username kudu)
I20260928 21:58:28.443675 24761 webserver.cc:460] Webserver started at http://127.2.182.126:35665/ using document root <none> and password file <none>
I20260928 21:58:28.443879 24761 fs_manager.cc:359] Metadata directory not provided
I20260928 21:58:28.443922 24761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:58:28.444018 24761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:58:28.445039 24761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632440055392-2777-0/minicluster-data/master-0/data/instance:
uuid: "c3a546982d4f4d9a9859f2f4fe0aef2e"
format_stamp: "Formatted at 2026-09-28 21:58:28 on dist-test-slave-k6lz"
server_key: "e59f964b01dd3320ef67b1f5ff6a24b6"
server_key_iv: "7a0cad975b7a2d34c5a922d61ab95635"
server_key_version: "encryptionkey@0"
I20260928 21:58:28.445317 24761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632440055392-2777-0/minicluster-data/master-0/wal/instance:
uuid: "c3a546982d4f4d9a9859f2f4fe0aef2e"
format_stamp: "Formatted at 2026-09-28 21:58:28 on dist-test-slave-k6lz"
server_key: "e59f964b01dd3320ef67b1f5ff6a24b6"
server_key_iv: "7a0cad975b7a2d34c5a922d61ab95635"
server_key_version: "encryptionkey@0"
I20260928 21:58:28.446530 24761 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:58:28.447252 24776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:28.447458 24761 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:58:28.447556 24761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632440055392-2777-0/minicluster-data/master-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632440055392-2777-0/minicluster-data/master-0/wal
uuid: "c3a546982d4f4d9a9859f2f4fe0aef2e"
format_stamp: "Formatted at 2026-09-28 21:58:28 on dist-test-slave-k6lz"
server_key: "e59f964b01dd3320ef67b1f5ff6a24b6"
server_key_iv: "7a0cad975b7a2d34c5a922d61ab95635"
server_key_version: "encryptionkey@0"
I20260928 21:58:28.447628 24761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632440055392-2777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632440055392-2777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632440055392-2777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:58:28.467896 24761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:58:28.469965 24761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:58:28.470080 24761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:58:28.473742 24761 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.126:34515
I20260928 21:58:28.473804 24828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.126:34515 every 8 connection(s)
I20260928 21:58:28.474090 24761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
I20260928 21:58:28.474973 24829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:58:28.477322 24829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3a546982d4f4d9a9859f2f4fe0aef2e: Bootstrap starting.
I20260928 21:58:28.477864 24829 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3a546982d4f4d9a9859f2f4fe0aef2e: Neither blocks nor log segments found. Creating new log.
I20260928 21:58:28.478120 24829 log.cc:824] T 00000000000000000000000000000000 P c3a546982d4f4d9a9859f2f4fe0aef2e: Log is configured to *not* fsync() on all Append() calls
I20260928 21:58:28.478792 24829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3a546982d4f4d9a9859f2f4fe0aef2e: No bootstrap required, opened a new log
I20260928 21:58:28.480525 24829 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3a546982d4f4d9a9859f2f4fe0aef2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3a546982d4f4d9a9859f2f4fe0aef2e" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 34515 } }
I20260928 21:58:28.480667 24829 raft_consensus.cc:374] T 00000000000000000000000000000000 P c3a546982d4f4d9a9859f2f4fe0aef2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:58:28.480701 24829 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3a546982d4f4d9a9859f2f4fe0aef2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3a546982d4f4d9a9859f2f4fe0aef2e, State: Initialized, Role: FOLLOWER
I20260928 21:58:28.480854 24829 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3a546982d4f4d9a9859f2f4fe0aef2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3a546982d4f4d9a9859f2f4fe0aef2e" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 34515 } }
I20260928 21:58:28.480931 24829 raft_consensus.cc:388] T 00000000000000000000000000000000 P c3a546982d4f4d9a9859f2f4fe0aef2e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:58:28.480963 24829 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3a546982d4f4d9a9859f2f4fe0aef2e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:58:28.481011 24829 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3a546982d4f4d9a9859f2f4fe0aef2e [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:58:28.481875 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 24761
I20260928 21:58:28.481890 24829 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3a546982d4f4d9a9859f2f4fe0aef2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3a546982d4f4d9a9859f2f4fe0aef2e" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 34515 } }
I20260928 21:58:28.481957 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632440055392-2777-0/minicluster-data/master-0/wal/instance
I20260928 21:58:28.481994 24829 leader_election.cc:302] T 00000000000000000000000000000000 P c3a546982d4f4d9a9859f2f4fe0aef2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3a546982d4f4d9a9859f2f4fe0aef2e; no voters:
I20260928 21:58:28.482142 2777 external_mini_cluster.cc:1428] Setting key cfb5bc612bf7190ac54d9bdfd5400e9c
I20260928 21:58:28.482157 24829 leader_election.cc:288] T 00000000000000000000000000000000 P c3a546982d4f4d9a9859f2f4fe0aef2e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:58:28.482190 24832 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3a546982d4f4d9a9859f2f4fe0aef2e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:58:28.482340 24832 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3a546982d4f4d9a9859f2f4fe0aef2e [term 1 LEADER]: Becoming Leader. State: Replica: c3a546982d4f4d9a9859f2f4fe0aef2e, State: Running, Role: LEADER
I20260928 21:58:28.482420 24829 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3a546982d4f4d9a9859f2f4fe0aef2e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:58:28.482456 24832 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3a546982d4f4d9a9859f2f4fe0aef2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3a546982d4f4d9a9859f2f4fe0aef2e" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 34515 } }
I20260928 21:58:28.483083 24834 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3a546982d4f4d9a9859f2f4fe0aef2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3a546982d4f4d9a9859f2f4fe0aef2e. Latest consensus state: current_term: 1 leader_uuid: "c3a546982d4f4d9a9859f2f4fe0aef2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3a546982d4f4d9a9859f2f4fe0aef2e" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 34515 } } }
I20260928 21:58:28.483095 24833 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3a546982d4f4d9a9859f2f4fe0aef2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c3a546982d4f4d9a9859f2f4fe0aef2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3a546982d4f4d9a9859f2f4fe0aef2e" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 34515 } } }
I20260928 21:58:28.483182 24834 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3a546982d4f4d9a9859f2f4fe0aef2e [sys.catalog]: This master's current role is: LEADER
I20260928 21:58:28.483194 24833 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3a546982d4f4d9a9859f2f4fe0aef2e [sys.catalog]: This master's current role is: LEADER
I20260928 21:58:28.483446 24843 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:58:28.484032 24843 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:58:28.485590 24843 catalog_manager.cc:1347] Generated new cluster ID: 275f308fba0f4a75811abfe2261207f6
I20260928 21:58:28.485643 24843 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:58:28.487618 24836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:28.482657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59873 (local address 127.2.182.126:34515)
0928 21:58:28.482818 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:28.482829 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:28.482865 (+ 36us) server_negotiation.cc:407] Connection header received
0928 21:58:28.482981 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:28.482989 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:28.483049 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:28.483458 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:28.483583 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:28.484986 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:28.485869 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:28.486281 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:28.486375 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:28.487010 (+ 635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:28.487014 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:28.487015 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:58:28.487186 (+ 171us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:28.487215 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:58:28.487225 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:28.487280 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:28.487377 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 21:58:28.487447 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":77,"threads_started":1}
I20260928 21:58:28.488423 2777 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:58:28.493954 24836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:28.489656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49715 (local address 127.2.182.126:34515)
0928 21:58:28.489690 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:28.489691 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:28.491541 (+ 1850us) server_negotiation.cc:407] Connection header received
0928 21:58:28.491581 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:28.491582 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:28.491592 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:28.491633 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:28.491759 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:28.492239 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:28.492942 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:28.493127 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:28.493179 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:28.493746 (+ 567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:28.493747 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:28.493748 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:58:28.493752 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:28.493765 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:58:28.493769 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:28.493813 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:28.493861 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 21:58:28.493884 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 21:58:28.502673 24843 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:58:28.503438 24843 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:58:28.520803 24843 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c3a546982d4f4d9a9859f2f4fe0aef2e: Generated new TSK 0
I20260928 21:58:28.521049 24843 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:58:28.531610 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 24761
2026-09-28T21:58:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1267 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632440055392-2777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:58:28 dist-test-slave-k6lz krb5kdc[24860](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:58:28 dist-test-slave-k6lz krb5kdc[24860](info): set up 2 sockets
Sep 28 21:58:28 dist-test-slave-k6lz krb5kdc[24860](info): commencing operation
krb5kdc: starting...
W20260928 21:58:29.491312 2777 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Sep 28 21:58:29 dist-test-slave-k6lz krb5kdc[24860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:58:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:58:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab.
Entry for principal kudu/127.2.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab.
I20260928 21:58:29.611052 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632440055392-2777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632440055392-2777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632440055392-2777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:41837
--webserver_interface=127.2.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab
--principal=kudu/127.2.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:37073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.126:41837
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:58:29.692842 24873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:58:29.693066 24873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:58:29.693089 24873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:58:29.694298 24873 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:58:29.694350 24873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:58:29.694363 24873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:58:29.694375 24873 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:58:29.694386 24873 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:58:29.695988 24873 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:37073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632440055392-2777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632440055392-2777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.126:41837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab
--principal=kudu/127.2.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:41837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24873
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632440055392-2777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:58:29.696238 24873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:58:29.696448 24873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:58:29.702427 24879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:29.702415 24881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:29.702581 24873 server_base.cc:1034] running on GCE node
W20260928 21:58:29.702430 24878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:29.703123 24873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:58:29.703482 24873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:58:29.704738 24873 hybrid_clock.cc:648] HybridClock initialized: now 1790632709704703 us; error 53 us; skew 500 ppm
Sep 28 21:58:29 dist-test-slave-k6lz krb5kdc[24860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632709, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:58:29.707578 24873 init.cc:377] Logged in from keytab as kudu/127.2.182.126@KRBTEST.COM (short username kudu)
I20260928 21:58:29.708025 24873 webserver.cc:460] Webserver started at http://127.2.182.126:38817/ using document root <none> and password file <none>
I20260928 21:58:29.708230 24873 fs_manager.cc:359] Metadata directory not provided
I20260928 21:58:29.708279 24873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:58:29.708405 24873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:58:29.709570 24873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632440055392-2777-0/minicluster-data/master-0/data/instance:
uuid: "6de487d0eb4a4b9ead4e492797d9a012"
format_stamp: "Formatted at 2026-09-28 21:58:29 on dist-test-slave-k6lz"
server_key: "3a7cf9ff3a3f909473ed408a07820f62"
server_key_iv: "80e9e62ad7fad2e097ca0fb5c008791d"
server_key_version: "encryptionkey@0"
I20260928 21:58:29.709921 24873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632440055392-2777-0/minicluster-data/master-0/wal/instance:
uuid: "6de487d0eb4a4b9ead4e492797d9a012"
format_stamp: "Formatted at 2026-09-28 21:58:29 on dist-test-slave-k6lz"
server_key: "3a7cf9ff3a3f909473ed408a07820f62"
server_key_iv: "80e9e62ad7fad2e097ca0fb5c008791d"
server_key_version: "encryptionkey@0"
I20260928 21:58:29.711201 24873 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:58:29.712064 24888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:29.712301 24873 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:58:29.712404 24873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632440055392-2777-0/minicluster-data/master-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632440055392-2777-0/minicluster-data/master-0/wal
uuid: "6de487d0eb4a4b9ead4e492797d9a012"
format_stamp: "Formatted at 2026-09-28 21:58:29 on dist-test-slave-k6lz"
server_key: "3a7cf9ff3a3f909473ed408a07820f62"
server_key_iv: "80e9e62ad7fad2e097ca0fb5c008791d"
server_key_version: "encryptionkey@0"
I20260928 21:58:29.712486 24873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632440055392-2777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632440055392-2777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632440055392-2777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:58:29.716008 24873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:58:29.718149 24873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:58:29.718286 24873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:58:29.722292 24873 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.126:41837
I20260928 21:58:29.722324 24940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.126:41837 every 8 connection(s)
I20260928 21:58:29.722642 24873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
I20260928 21:58:29.723641 24941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:58:29.726019 24941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6de487d0eb4a4b9ead4e492797d9a012: Bootstrap starting.
I20260928 21:58:29.726262 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 24873
I20260928 21:58:29.726384 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632440055392-2777-0/minicluster-data/master-0/wal/instance
I20260928 21:58:29.726559 2777 external_mini_cluster.cc:1428] Setting key 1056d3d51015babe59c76aa02da82548
I20260928 21:58:29.726897 24941 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6de487d0eb4a4b9ead4e492797d9a012: Neither blocks nor log segments found. Creating new log.
I20260928 21:58:29.727229 24941 log.cc:824] T 00000000000000000000000000000000 P 6de487d0eb4a4b9ead4e492797d9a012: Log is configured to *not* fsync() on all Append() calls
I20260928 21:58:29.728109 24941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6de487d0eb4a4b9ead4e492797d9a012: No bootstrap required, opened a new log
I20260928 21:58:29.729281 24944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:29.727161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50759 (local address 127.2.182.126:41837)
0928 21:58:29.727384 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:29.727394 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:29.727424 (+ 30us) server_negotiation.cc:407] Connection header received
0928 21:58:29.727913 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:29.727920 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:29.727947 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:29.728077 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:29.728770 (+ 693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:29.728773 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:29.728773 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:58:29.728778 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:29.728817 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:58:29.728828 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:29.728911 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:29.729023 (+ 112us) server_negotiation.cc:299] Negotiation successful
0928 21:58:29.729076 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":68,"threads_started":1}
I20260928 21:58:29.730430 2777 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:58:29.730474 24941 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6de487d0eb4a4b9ead4e492797d9a012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6de487d0eb4a4b9ead4e492797d9a012" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 41837 } }
I20260928 21:58:29.730681 24941 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6de487d0eb4a4b9ead4e492797d9a012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:58:29.730717 24941 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6de487d0eb4a4b9ead4e492797d9a012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6de487d0eb4a4b9ead4e492797d9a012, State: Initialized, Role: FOLLOWER
I20260928 21:58:29.730834 24941 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6de487d0eb4a4b9ead4e492797d9a012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6de487d0eb4a4b9ead4e492797d9a012" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 41837 } }
I20260928 21:58:29.730916 24941 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6de487d0eb4a4b9ead4e492797d9a012 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:58:29.730953 24941 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6de487d0eb4a4b9ead4e492797d9a012 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:58:29.730998 24941 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6de487d0eb4a4b9ead4e492797d9a012 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:58:29.731832 24941 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6de487d0eb4a4b9ead4e492797d9a012 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6de487d0eb4a4b9ead4e492797d9a012" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 41837 } }
I20260928 21:58:29.731947 24941 leader_election.cc:302] T 00000000000000000000000000000000 P 6de487d0eb4a4b9ead4e492797d9a012 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6de487d0eb4a4b9ead4e492797d9a012; no voters:
I20260928 21:58:29.732105 24941 leader_election.cc:288] T 00000000000000000000000000000000 P 6de487d0eb4a4b9ead4e492797d9a012 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:58:29.732148 24950 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6de487d0eb4a4b9ead4e492797d9a012 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:58:29.732303 24950 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6de487d0eb4a4b9ead4e492797d9a012 [term 1 LEADER]: Becoming Leader. State: Replica: 6de487d0eb4a4b9ead4e492797d9a012, State: Running, Role: LEADER
I20260928 21:58:29.732352 24941 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6de487d0eb4a4b9ead4e492797d9a012 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:58:29.732470 24950 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6de487d0eb4a4b9ead4e492797d9a012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6de487d0eb4a4b9ead4e492797d9a012" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 41837 } }
I20260928 21:58:29.732965 24953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6de487d0eb4a4b9ead4e492797d9a012 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6de487d0eb4a4b9ead4e492797d9a012" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6de487d0eb4a4b9ead4e492797d9a012" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 41837 } } }
I20260928 21:58:29.732965 24952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6de487d0eb4a4b9ead4e492797d9a012 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6de487d0eb4a4b9ead4e492797d9a012. Latest consensus state: current_term: 1 leader_uuid: "6de487d0eb4a4b9ead4e492797d9a012" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6de487d0eb4a4b9ead4e492797d9a012" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 41837 } } }
I20260928 21:58:29.733191 24952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6de487d0eb4a4b9ead4e492797d9a012 [sys.catalog]: This master's current role is: LEADER
I20260928 21:58:29.733148 24953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6de487d0eb4a4b9ead4e492797d9a012 [sys.catalog]: This master's current role is: LEADER
I20260928 21:58:29.733420 24944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:29.732077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54765 (local address 127.2.182.126:41837)
0928 21:58:29.732117 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:29.732119 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:29.732262 (+ 143us) server_negotiation.cc:407] Connection header received
0928 21:58:29.732351 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:29.732352 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:29.732357 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:29.732399 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:29.733096 (+ 697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:29.733098 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:29.733098 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:58:29.733102 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:29.733118 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:58:29.733122 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:29.733167 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:29.733347 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 21:58:29.733360 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 21:58:29.733997 24960 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:58:29.734442 24960 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:58:29.735893 24960 catalog_manager.cc:1347] Generated new cluster ID: 61244254aeea410ab0cd56178e55b1a6
I20260928 21:58:29.735942 24960 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:58:29.743954 24960 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:58:29.744179 24960 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:58:29.747835 24960 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6de487d0eb4a4b9ead4e492797d9a012: Generated new TSK 0
I20260928 21:58:29.748078 24960 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:58:29.758459 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 24873
2026-09-28T21:58:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1227 ms)
[----------] 18 tests from AuthTokenIssuingTest (22315 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632440055392-2777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:58:29 dist-test-slave-k6lz krb5kdc[24972](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:58:29 dist-test-slave-k6lz krb5kdc[24972](info): set up 2 sockets
Sep 28 21:58:29 dist-test-slave-k6lz krb5kdc[24972](info): commencing operation
krb5kdc: starting...
W20260928 21:58:30.728160 2777 mini_kdc.cc:121] Time spent starting KDC: real 0.952s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Sep 28 21:58:30 dist-test-slave-k6lz krb5kdc[24972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632710, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:58:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:58:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab.
Entry for principal kudu/127.2.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab.
I20260928 21:58:30.855993 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632440055392-2777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632440055392-2777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632440055392-2777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:45245
--webserver_interface=127.2.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab
--principal=kudu/127.2.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:41141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.126:45245,127.2.182.125:46575,127.2.182.124:43045
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:58:30.936046 24985 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 21:58:30.936285 24985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:58:30.936301 24985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:58:30.936321 24985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:58:30.937557 24985 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:58:30.937610 24985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:58:30.937623 24985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:58:30.937642 24985 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:58:30.937654 24985 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:58:30.939226 24985 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:41141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632440055392-2777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632440055392-2777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.126:45245,127.2.182.125:46575,127.2.182.124:43045
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab
--principal=kudu/127.2.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:45245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24985
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632440055392-2777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:58:30.939502 24985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:58:30.939738 24985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:58:30.944707 24993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:30.944793 24985 server_base.cc:1034] running on GCE node
W20260928 21:58:30.944768 24990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:30.944660 24991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:30.945135 24985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:58:30.945413 24985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:58:30.946638 24985 hybrid_clock.cc:648] HybridClock initialized: now 1790632710946596 us; error 54 us; skew 500 ppm
Sep 28 21:58:30 dist-test-slave-k6lz krb5kdc[24972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632710, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:58:30.949220 24985 init.cc:377] Logged in from keytab as kudu/127.2.182.126@KRBTEST.COM (short username kudu)
I20260928 21:58:30.949604 24985 webserver.cc:460] Webserver started at http://127.2.182.126:46107/ using document root <none> and password file <none>
I20260928 21:58:30.949797 24985 fs_manager.cc:359] Metadata directory not provided
I20260928 21:58:30.949836 24985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:58:30.949924 24985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:58:30.950947 24985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632440055392-2777-0/minicluster-data/master-0/data/instance:
uuid: "d5ae094ccd084b18bbf7523adb98e95c"
format_stamp: "Formatted at 2026-09-28 21:58:30 on dist-test-slave-k6lz"
server_key: "520872dc87372b88027fbf3e411de3a4"
server_key_iv: "392345edfd8619dcbe1b6b47ee3f4830"
server_key_version: "encryptionkey@0"
I20260928 21:58:30.951243 24985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632440055392-2777-0/minicluster-data/master-0/wal/instance:
uuid: "d5ae094ccd084b18bbf7523adb98e95c"
format_stamp: "Formatted at 2026-09-28 21:58:30 on dist-test-slave-k6lz"
server_key: "520872dc87372b88027fbf3e411de3a4"
server_key_iv: "392345edfd8619dcbe1b6b47ee3f4830"
server_key_version: "encryptionkey@0"
I20260928 21:58:30.952466 24985 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:58:30.953178 25000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:30.953349 24985 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:58:30.953418 24985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632440055392-2777-0/minicluster-data/master-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632440055392-2777-0/minicluster-data/master-0/wal
uuid: "d5ae094ccd084b18bbf7523adb98e95c"
format_stamp: "Formatted at 2026-09-28 21:58:30 on dist-test-slave-k6lz"
server_key: "520872dc87372b88027fbf3e411de3a4"
server_key_iv: "392345edfd8619dcbe1b6b47ee3f4830"
server_key_version: "encryptionkey@0"
I20260928 21:58:30.953486 24985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632440055392-2777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632440055392-2777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632440055392-2777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:58:30.960687 24985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:58:30.962695 24985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:58:30.962802 24985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:58:30.966573 24985 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.126:45245
I20260928 21:58:30.966638 25052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.126:45245 every 8 connection(s)
I20260928 21:58:30.966923 24985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
I20260928 21:58:30.967983 25053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:58:30.969713 25053 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.182.126" port: 45245 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:58:30.970649 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 24985
I20260928 21:58:30.970754 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632440055392-2777-0/minicluster-data/master-0/wal/instance
I20260928 21:58:30.970922 2777 external_mini_cluster.cc:1428] Setting key 782258f6ad1d01a2285595146b37c98e
WARNING: no policy specified for kudu/127.2.182.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.125@KRBTEST.COM" created.
Sep 28 21:58:30 dist-test-slave-k6lz krb5kdc[24972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632710, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.126@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:58:30.986994 25055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:30.970455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:45245 (local address 127.0.0.1:59300)
0928 21:58:30.970752 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:30.970849 (+ 97us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:30.971410 (+ 561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:30.971677 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:30.971679 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:30.971979 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:58:30.972214 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:30.972220 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:30.973549 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:30.973549 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:30.974280 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:30.974283 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:30.975042 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:30.975044 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:30.975103 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:30.981598 (+ 6495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:58:30.981614 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:58:30.983292 (+ 1678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:58:30.985269 (+ 1977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:30.985276 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:30.985278 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:30.985569 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:30.985853 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:30.985853 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:30.985854 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:30.985944 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:30.986410 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:58:30.986416 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:58:30.986480 (+ 64us) client_negotiation.cc:769] Sending connection context
0928 21:58:30.986628 (+ 148us) client_negotiation.cc:240] Negotiation successful
0928 21:58:30.986776 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":201,"threads_started":1}
I20260928 21:58:30.987002 25056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:30.970528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59300 (local address 127.2.182.126:45245)
0928 21:58:30.970827 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:30.970838 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:30.970951 (+ 113us) server_negotiation.cc:407] Connection header received
0928 21:58:30.971569 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:30.971571 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:30.971588 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:30.971664 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:30.972312 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:30.973438 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:30.974369 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:30.974878 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:30.975000 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:30.983460 (+ 8460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:30.983473 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:30.983473 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:30.983482 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:30.985165 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:30.985657 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:30.985662 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:30.985662 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:30.985776 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:30.986007 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:30.986008 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:30.986008 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:30.986331 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:30.986419 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:30.986684 (+ 265us) server_negotiation.cc:299] Negotiation successful
0928 21:58:30.986855 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":219,"thread_start_us":152,"threads_started":1}
I20260928 21:58:30.987784 25053 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.182.125" port: 46575 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:58:30.988109 25055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:30.988007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.125:46575 (local address 127.0.0.1:42520)
0928 21:58:30.988050 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:30.988090 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.125:46575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 21:58:30.988229 25001 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.182.125:46575: connect: Connection refused (error 111)
I20260928 21:58:30.988679 25055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:30.988592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.125:46575 (local address 127.0.0.1:42530)
0928 21:58:30.988633 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:30.988660 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.125:46575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 21:58:30.988828 25053 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.182.125:46575: Network error: Client connection negotiation failed: client connection to 127.2.182.125:46575: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.125.keytab.
Entry for principal kudu/127.2.182.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.125.keytab.
I20260928 21:58:30.998358 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632440055392-2777-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632440055392-2777-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632440055392-2777-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632440055392-2777-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.125:46575
--webserver_interface=127.2.182.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.125.keytab
--principal=kudu/127.2.182.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:41141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.126:45245,127.2.182.125:46575,127.2.182.124:43045
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:58:31.038062 25053 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.182.125" port: 46575 } attempt: 1
I20260928 21:58:31.038645 25055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:31.038420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.125:46575 (local address 127.0.0.1:42534)
0928 21:58:31.038577 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:31.038614 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.125:46575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
I20260928 21:58:31.038971 25055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:31.038885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.125:46575 (local address 127.0.0.1:42538)
0928 21:58:31.038922 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:31.038952 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.125:46575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260928 21:58:31.039059 25053 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.182.125:46575: Network error: Client connection negotiation failed: client connection to 127.2.182.125:46575: connect: Connection refused (error 111)
W20260928 21:58:31.081300 25060 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 21:58:31.081532 25060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:58:31.081557 25060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:58:31.081583 25060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:58:31.082769 25060 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:58:31.082815 25060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:58:31.082834 25060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:58:31.082849 25060 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:58:31.082864 25060 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:58:31.084415 25060 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:41141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632440055392-2777-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632440055392-2777-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.126:45245,127.2.182.125:46575,127.2.182.124:43045
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.125.keytab
--principal=kudu/127.2.182.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.125:46575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632440055392-2777-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.182.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25060
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632440055392-2777-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:58:31.084674 25060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:58:31.084892 25060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:58:31.090248 25068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:31.090305 25060 server_base.cc:1034] running on GCE node
W20260928 21:58:31.090255 25065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:31.090269 25066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:31.090796 25060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:58:31.091101 25060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:58:31.092271 25060 hybrid_clock.cc:648] HybridClock initialized: now 1790632711092258 us; error 24 us; skew 500 ppm
Sep 28 21:58:31 dist-test-slave-k6lz krb5kdc[24972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632711, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:58:31.095288 25060 init.cc:377] Logged in from keytab as kudu/127.2.182.125@KRBTEST.COM (short username kudu)
I20260928 21:58:31.095901 25060 webserver.cc:460] Webserver started at http://127.2.182.125:45079/ using document root <none> and password file <none>
I20260928 21:58:31.096217 25060 fs_manager.cc:359] Metadata directory not provided
I20260928 21:58:31.096357 25060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:58:31.096506 25060 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:58:31.097815 25060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632440055392-2777-0/minicluster-data/master-1/data/instance:
uuid: "67a951337146439ba05102c122982ca2"
format_stamp: "Formatted at 2026-09-28 21:58:31 on dist-test-slave-k6lz"
server_key: "4f99563df4360fdd6e19f9bb7f0232d6"
server_key_iv: "43f4cab7b27a48d1c2b65ec47b1ffe41"
server_key_version: "encryptionkey@0"
I20260928 21:58:31.098143 25060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632440055392-2777-0/minicluster-data/master-1/wal/instance:
uuid: "67a951337146439ba05102c122982ca2"
format_stamp: "Formatted at 2026-09-28 21:58:31 on dist-test-slave-k6lz"
server_key: "4f99563df4360fdd6e19f9bb7f0232d6"
server_key_iv: "43f4cab7b27a48d1c2b65ec47b1ffe41"
server_key_version: "encryptionkey@0"
I20260928 21:58:31.099431 25060 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:58:31.100263 25075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:31.100422 25060 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:58:31.100503 25060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632440055392-2777-0/minicluster-data/master-1/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632440055392-2777-0/minicluster-data/master-1/wal
uuid: "67a951337146439ba05102c122982ca2"
format_stamp: "Formatted at 2026-09-28 21:58:31 on dist-test-slave-k6lz"
server_key: "4f99563df4360fdd6e19f9bb7f0232d6"
server_key_iv: "43f4cab7b27a48d1c2b65ec47b1ffe41"
server_key_version: "encryptionkey@0"
I20260928 21:58:31.100564 25060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632440055392-2777-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632440055392-2777-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632440055392-2777-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:58:31.107229 25053 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.182.125" port: 46575 } attempt: 2
I20260928 21:58:31.107810 25055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:31.107681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.125:46575 (local address 127.0.0.1:42554)
0928 21:58:31.107735 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:31.107779 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.125:46575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 21:58:31.108153 25055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:31.108079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.125:46575 (local address 127.0.0.1:42556)
0928 21:58:31.108111 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:31.108136 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.125:46575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260928 21:58:31.108232 25053 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.182.125:46575: Network error: Client connection negotiation failed: client connection to 127.2.182.125:46575: connect: Connection refused (error 111)
I20260928 21:58:31.117014 25060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:58:31.119226 25060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:58:31.119390 25060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:58:31.123672 25060 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.125:46575
I20260928 21:58:31.123720 25127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.125:46575 every 8 connection(s)
I20260928 21:58:31.124056 25060 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632440055392-2777-0/minicluster-data/master-1/data/info.pb
I20260928 21:58:31.124964 25128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:58:31.126413 25128 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.182.126" port: 45245 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 21:58:31 dist-test-slave-k6lz krb5kdc[24972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632711, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.125@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:58:31.133769 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 25060
I20260928 21:58:31.133885 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632440055392-2777-0/minicluster-data/master-1/wal/instance
I20260928 21:58:31.134076 2777 external_mini_cluster.cc:1428] Setting key 65b37c17de1c25f74433d391552818fc
I20260928 21:58:31.135634 25056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:31.127141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59316 (local address 127.2.182.126:45245)
0928 21:58:31.127235 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:31.127237 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:31.127322 (+ 85us) server_negotiation.cc:407] Connection header received
0928 21:58:31.127870 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:31.127871 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:31.127885 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:31.127951 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:31.128670 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:31.129199 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:31.130180 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:31.130377 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:31.130434 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:31.132523 (+ 2089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:31.132525 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:31.132525 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:31.132532 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:31.133950 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:31.134439 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:31.134441 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:31.134441 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:31.134508 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:31.134767 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:31.134768 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:31.134768 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:31.134899 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:31.134967 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:31.135313 (+ 346us) server_negotiation.cc:299] Negotiation successful
0928 21:58:31.135541 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 21:58:31.135695 25130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:31.127051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:45245 (local address 127.0.0.1:59316)
0928 21:58:31.127213 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:31.127255 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:31.127758 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:31.128070 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:31.128074 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:31.128356 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:58:31.128565 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:31.128570 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:31.129284 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:31.129285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:31.130085 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:31.130097 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:31.130471 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:31.130472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:31.130582 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:31.131131 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:58:31.131139 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:58:31.132441 (+ 1302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:58:31.134090 (+ 1649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:31.134097 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:31.134100 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:31.134357 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:31.134592 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:31.134593 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:31.134594 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:31.134708 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:31.135032 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:58:31.135037 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:58:31.135166 (+ 129us) client_negotiation.cc:769] Sending connection context
0928 21:58:31.135263 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 21:58:31.135454 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":77,"threads_started":1}
I20260928 21:58:31.136397 25128 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.182.125" port: 46575 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 21:58:31 dist-test-slave-k6lz krb5kdc[24972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632711, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.125@KRBTEST.COM for kudu/127.2.182.125@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.182.124@KRBTEST.COM; defaulting to no policy
I20260928 21:58:31.144320 25130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:31.136723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.125:46575 (local address 127.0.0.1:42572)
0928 21:58:31.136772 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:31.136783 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:31.136843 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:31.137065 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:31.137066 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:31.137182 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:58:31.137236 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:31.137238 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:31.138491 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:31.138492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:31.139004 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:31.139005 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:31.139692 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:31.139693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:31.139752 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:31.140156 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:58:31.140160 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:58:31.141425 (+ 1265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:58:31.143196 (+ 1771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:31.143197 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:31.143197 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:31.143403 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:31.143640 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:31.143640 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:31.143641 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:31.143695 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:31.143954 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:58:31.143955 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:58:31.144010 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 21:58:31.144080 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 21:58:31.144181 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 21:58:31.144393 25132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:31.136763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42572 (local address 127.2.182.125:46575)
0928 21:58:31.136921 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:31.136926 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:31.136948 (+ 22us) server_negotiation.cc:407] Connection header received
0928 21:58:31.136969 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:31.136971 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:31.136985 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:31.137059 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:31.137301 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:31.138397 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:31.139089 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:31.139606 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:31.139693 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:31.141533 (+ 1840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:31.141543 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:31.141544 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:31.141552 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:31.143099 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:31.143481 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:31.143485 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:31.143486 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:31.143545 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:31.143741 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:31.143741 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:31.143741 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:31.143884 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:31.143941 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:31.144106 (+ 165us) server_negotiation.cc:299] Negotiation successful
0928 21:58:31.144242 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":130,"thread_start_us":94,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.124@KRBTEST.COM" created.
I20260928 21:58:31.145149 25128 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.182.124" port: 43045 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:58:31.145598 25130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:31.145469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.124:43045 (local address 127.0.0.1:57836)
0928 21:58:31.145517 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:31.145572 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.124:43045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 21:58:31.145759 25076 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.182.124:43045: connect: Connection refused (error 111)
I20260928 21:58:31.146201 25130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:31.146092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.124:43045 (local address 127.0.0.1:57850)
0928 21:58:31.146139 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:31.146179 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.124:43045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 21:58:31.146368 25128 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.182.124:43045: Network error: Client connection negotiation failed: client connection to 127.2.182.124:43045: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.124.keytab.
Entry for principal kudu/127.2.182.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.124.keytab.
I20260928 21:58:31.163368 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632440055392-2777-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632440055392-2777-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632440055392-2777-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632440055392-2777-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.124:43045
--webserver_interface=127.2.182.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.124.keytab
--principal=kudu/127.2.182.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:41141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.126:45245,127.2.182.125:46575,127.2.182.124:43045
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:58:31.195566 25128 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.182.124" port: 43045 } attempt: 1
I20260928 21:58:31.196029 25130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:31.195918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.124:43045 (local address 127.0.0.1:57864)
0928 21:58:31.195971 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:31.196002 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.124:43045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 21:58:31.196329 25130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:31.196254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.124:43045 (local address 127.0.0.1:57868)
0928 21:58:31.196289 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:31.196312 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.124:43045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260928 21:58:31.196417 25128 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.182.124:43045: Network error: Client connection negotiation failed: client connection to 127.2.182.124:43045: connect: Connection refused (error 111)
I20260928 21:58:31.199376 25053 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.182.125" port: 46575 } attempt: 3
Sep 28 21:58:31 dist-test-slave-k6lz krb5kdc[24972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632710, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.126@KRBTEST.COM for kudu/127.2.182.125@KRBTEST.COM
I20260928 21:58:31.205928 25055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:31.199693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.125:46575 (local address 127.0.0.1:42584)
0928 21:58:31.199745 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:31.199755 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:31.199803 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:31.199913 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:31.199914 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:31.200071 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:58:31.200141 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:31.200143 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:31.200769 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:31.200770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:31.201329 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:31.201331 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:31.201727 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:31.201728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:31.201804 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:31.202259 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:58:31.202262 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:58:31.203532 (+ 1270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:58:31.205002 (+ 1470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:31.205003 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:31.205003 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:31.205176 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:31.205342 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:31.205343 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:31.205343 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:31.205400 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:31.205615 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:58:31.205615 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:58:31.205663 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 21:58:31.205722 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 21:58:31.205829 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 21:58:31.205928 25132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:31.199757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42584 (local address 127.2.182.125:46575)
0928 21:58:31.199804 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:31.199806 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:31.199831 (+ 25us) server_negotiation.cc:407] Connection header received
0928 21:58:31.199857 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:31.199857 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:31.199869 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:31.199911 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:31.200211 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:31.200697 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:31.201409 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:31.201648 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:31.201724 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:31.203619 (+ 1895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:31.203620 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:31.203621 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:31.203629 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:31.204920 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:31.205241 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:31.205242 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:31.205242 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:31.205291 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:31.205448 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:31.205449 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:31.205449 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:31.205554 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:31.205607 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:31.205751 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 21:58:31.205831 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 21:58:31.206321 25053 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.182.124" port: 43045 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:58:31.206636 25055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:31.206542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.124:43045 (local address 127.0.0.1:57876)
0928 21:58:31.206584 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:31.206618 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.124:43045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 21:58:31.206875 25055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:31.206812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.124:43045 (local address 127.0.0.1:57882)
0928 21:58:31.206839 (+ 27us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:31.206861 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.124:43045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260928 21:58:31.206935 25053 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.182.124:43045: Network error: Client connection negotiation failed: client connection to 127.2.182.124:43045: connect: Connection refused (error 111)
I20260928 21:58:31.238098 25053 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.182.124" port: 43045 } attempt: 1
I20260928 21:58:31.238647 25055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:31.238452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.124:43045 (local address 127.0.0.1:57886)
0928 21:58:31.238506 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:31.238544 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.124:43045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 21:58:31.239164 25055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:31.239032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.124:43045 (local address 127.0.0.1:57890)
0928 21:58:31.239106 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:31.239132 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.124:43045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260928 21:58:31.239269 25053 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.182.124:43045: Network error: Client connection negotiation failed: client connection to 127.2.182.124:43045: connect: Connection refused (error 111)
W20260928 21:58:31.246076 25135 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 21:58:31.246306 25135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:58:31.246331 25135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:58:31.246359 25135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:58:31.247606 25135 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:58:31.247649 25135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:58:31.247669 25135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:58:31.247684 25135 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:58:31.247701 25135 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:58:31.249287 25135 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:41141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632440055392-2777-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632440055392-2777-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.126:45245,127.2.182.125:46575,127.2.182.124:43045
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.124.keytab
--principal=kudu/127.2.182.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.124:43045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632440055392-2777-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.182.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25135
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632440055392-2777-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:58:31.249552 25135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:58:31.249768 25135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:58:31.254315 25143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:31.254535 25141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:31.254644 25140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:31.254968 25135 server_base.cc:1034] running on GCE node
I20260928 21:58:31.255229 25135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:58:31.255616 25135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:58:31.256793 25135 hybrid_clock.cc:648] HybridClock initialized: now 1790632711256775 us; error 34 us; skew 500 ppm
Sep 28 21:58:31 dist-test-slave-k6lz krb5kdc[24972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632711, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:58:31.259410 25135 init.cc:377] Logged in from keytab as kudu/127.2.182.124@KRBTEST.COM (short username kudu)
I20260928 21:58:31.259970 25135 webserver.cc:460] Webserver started at http://127.2.182.124:42561/ using document root <none> and password file <none>
I20260928 21:58:31.260167 25135 fs_manager.cc:359] Metadata directory not provided
I20260928 21:58:31.260213 25135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:58:31.260337 25135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:58:31.261400 25135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632440055392-2777-0/minicluster-data/master-2/data/instance:
uuid: "c52ef9dde5b84823ba2b87d8f42123c5"
format_stamp: "Formatted at 2026-09-28 21:58:31 on dist-test-slave-k6lz"
server_key: "bc501e5a61ab10a5b7f56122478a95a1"
server_key_iv: "d48f856de8a79c516a0978fb5928fca0"
server_key_version: "encryptionkey@0"
I20260928 21:58:31.261765 25135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632440055392-2777-0/minicluster-data/master-2/wal/instance:
uuid: "c52ef9dde5b84823ba2b87d8f42123c5"
format_stamp: "Formatted at 2026-09-28 21:58:31 on dist-test-slave-k6lz"
server_key: "bc501e5a61ab10a5b7f56122478a95a1"
server_key_iv: "d48f856de8a79c516a0978fb5928fca0"
server_key_version: "encryptionkey@0"
I20260928 21:58:31.263073 25135 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:58:31.263973 25150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:31.264180 25135 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:58:31.264264 25135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632440055392-2777-0/minicluster-data/master-2/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632440055392-2777-0/minicluster-data/master-2/wal
uuid: "c52ef9dde5b84823ba2b87d8f42123c5"
format_stamp: "Formatted at 2026-09-28 21:58:31 on dist-test-slave-k6lz"
server_key: "bc501e5a61ab10a5b7f56122478a95a1"
server_key_iv: "d48f856de8a79c516a0978fb5928fca0"
server_key_version: "encryptionkey@0"
I20260928 21:58:31.264334 25135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632440055392-2777-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632440055392-2777-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632440055392-2777-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:58:31.264540 25128 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.182.124" port: 43045 } attempt: 2
I20260928 21:58:31.264922 25130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:31.264810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.124:43045 (local address 127.0.0.1:57902)
0928 21:58:31.264862 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:31.264893 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.124:43045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 21:58:31.265367 25130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:31.265284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.124:43045 (local address 127.0.0.1:57904)
0928 21:58:31.265326 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:31.265350 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.124:43045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 21:58:31.265450 25128 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.182.124:43045: Network error: Client connection negotiation failed: client connection to 127.2.182.124:43045: connect: Connection refused (error 111)
I20260928 21:58:31.272989 25135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:58:31.274958 25135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:58:31.275068 25135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:58:31.278836 25135 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.124:43045
I20260928 21:58:31.278867 25202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.124:43045 every 8 connection(s)
I20260928 21:58:31.279188 25135 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632440055392-2777-0/minicluster-data/master-2/data/info.pb
I20260928 21:58:31.280123 25203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:58:31.281733 25203 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.182.126" port: 45245 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 21:58:31 dist-test-slave-k6lz krb5kdc[24972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632711, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.124@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:58:31.288199 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 25135
I20260928 21:58:31.288300 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632440055392-2777-0/minicluster-data/master-2/wal/instance
I20260928 21:58:31.288458 2777 external_mini_cluster.cc:1428] Setting key 967a34704b813a8f9ddf4b086da0bf8b
I20260928 21:58:31.291056 25056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:31.282457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59318 (local address 127.2.182.126:45245)
0928 21:58:31.282538 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:31.282540 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:31.282709 (+ 169us) server_negotiation.cc:407] Connection header received
0928 21:58:31.283204 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:31.283205 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:31.283216 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:31.283269 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:31.283984 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:31.284577 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:31.285681 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:31.285868 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:31.285919 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:31.288371 (+ 2452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:31.288373 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:31.288373 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:31.288381 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:31.289767 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:31.290154 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:31.290155 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:31.290155 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:31.290212 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:31.290426 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:31.290427 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:31.290427 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:31.290553 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:31.290610 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:31.290876 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 21:58:31.290959 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 21:58:31.291225 25205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:31.282380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:45245 (local address 127.0.0.1:59318)
0928 21:58:31.282543 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:31.282595 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:31.283096 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:31.283405 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:31.283411 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:31.283676 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:58:31.283870 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:31.283875 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:31.284718 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:31.284719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:31.285566 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:31.285569 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:31.285958 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:31.285959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:31.286099 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:31.286753 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:58:31.286762 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:58:31.288271 (+ 1509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:58:31.289843 (+ 1572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:31.289851 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:31.289853 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:31.290087 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:31.290264 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:31.290264 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:31.290264 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:31.290371 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:31.290606 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:58:31.290612 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:58:31.290751 (+ 139us) client_negotiation.cc:769] Sending connection context
0928 21:58:31.290837 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 21:58:31.291006 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":64,"threads_started":1}
I20260928 21:58:31.292137 25203 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.182.125" port: 46575 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 21:58:31 dist-test-slave-k6lz krb5kdc[24972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632710, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.182.124@KRBTEST.COM
Sep 28 21:58:31 dist-test-slave-k6lz krb5kdc[24972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632711, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.124@KRBTEST.COM for kudu/127.2.182.125@KRBTEST.COM
I20260928 21:58:31.297168 25206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:31.288841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57916 (local address 127.2.182.124:43045)
0928 21:58:31.289005 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:31.289010 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:31.289130 (+ 120us) server_negotiation.cc:407] Connection header received
0928 21:58:31.289175 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:31.289178 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:31.289193 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:31.289256 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:31.289808 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:31.290497 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:31.291272 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:31.291739 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:31.291815 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:31.294394 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:31.294402 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:31.294402 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:31.294409 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:31.295901 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:31.296283 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:31.296286 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:31.296287 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:31.296339 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:31.296579 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:31.296580 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:31.296580 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:31.296767 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:31.296929 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:31.296985 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 21:58:31.297056 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":85,"threads_started":1}
I20260928 21:58:31.298779 25205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:31.292381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.125:46575 (local address 127.0.0.1:42586)
0928 21:58:31.292422 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:31.292433 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:31.292475 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:31.292613 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:31.292613 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:31.292718 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:58:31.292761 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:31.292762 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:31.293605 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:31.293606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:31.294289 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:31.294292 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:31.294699 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:31.294700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:31.294770 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:31.295038 (+ 268us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:58:31.295041 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:58:31.296186 (+ 1145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:58:31.297822 (+ 1636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:31.297823 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:31.297823 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:31.297989 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:31.298243 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:31.298244 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:31.298244 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:31.298302 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:31.298551 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:58:31.298551 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:58:31.298584 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 21:58:31.298628 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 21:58:31.298697 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 21:58:31.298823 25132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:31.292462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42586 (local address 127.2.182.125:46575)
0928 21:58:31.292509 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:31.292511 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:31.292519 (+ 8us) server_negotiation.cc:407] Connection header received
0928 21:58:31.292543 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:31.292544 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:31.292554 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:31.292605 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:31.292825 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:31.293506 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:31.294368 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:31.294604 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:31.294664 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:31.296279 (+ 1615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:31.296281 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:31.296281 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:31.296288 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:31.297748 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:31.298110 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:31.298111 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:31.298112 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:31.298179 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:31.298357 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:31.298358 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:31.298358 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:31.298492 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:31.298564 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:31.298653 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 21:58:31.298728 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 21:58:31.299292 25203 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.182.124" port: 43045 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 21:58:31 dist-test-slave-k6lz krb5kdc[24972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632710, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
Sep 28 21:58:31 dist-test-slave-k6lz krb5kdc[24972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632711, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.124@KRBTEST.COM for kudu/127.2.182.124@KRBTEST.COM
I20260928 21:58:31.306102 25056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:31.298076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59326 (local address 127.2.182.126:45245)
0928 21:58:31.298117 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:31.298119 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:31.298125 (+ 6us) server_negotiation.cc:407] Connection header received
0928 21:58:31.298147 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:31.298148 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:31.298158 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:31.298196 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:31.298763 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:31.299463 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:31.300635 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:31.300885 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:31.300934 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:31.303521 (+ 2587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:31.303523 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:31.303524 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:31.303532 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:31.305044 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:31.305375 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:31.305376 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:31.305377 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:31.305436 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:31.305609 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:31.305610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:31.305610 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:31.305724 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:31.305890 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:31.305924 (+ 34us) server_negotiation.cc:299] Negotiation successful
0928 21:58:31.306011 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 21:58:31.307139 25205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:31.300066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.124:43045 (local address 127.0.0.1:57926)
0928 21:58:31.300227 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:31.300244 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:31.300298 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:31.300445 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:31.300445 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:31.300650 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:58:31.300685 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:31.300687 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:31.301427 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:31.301428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:31.301889 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:31.301890 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:31.302342 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:31.302343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:31.302400 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:31.302766 (+ 366us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:58:31.302770 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:58:31.304036 (+ 1266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:58:31.306199 (+ 2163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:31.306200 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:31.306201 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:31.306388 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:31.306588 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:31.306588 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:31.306589 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:31.306645 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:31.306902 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:58:31.306903 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:58:31.306948 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 21:58:31.307004 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 21:58:31.307077 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260928 21:58:31.307238 25206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:31.300313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57926 (local address 127.2.182.124:43045)
0928 21:58:31.300352 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:31.300354 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:31.300360 (+ 6us) server_negotiation.cc:407] Connection header received
0928 21:58:31.300376 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:31.300376 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:31.300385 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:31.300429 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:31.300778 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:31.301350 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:31.301967 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:31.302267 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:31.302320 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:31.304122 (+ 1802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:31.304123 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:31.304124 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:31.304131 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:31.305542 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:31.306455 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:31.306457 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:31.306457 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:31.306522 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:31.306708 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:31.306709 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:31.306709 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:31.306843 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:31.306903 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:31.307043 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 21:58:31.307147 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":20}
I20260928 21:58:31.308725 25203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c52ef9dde5b84823ba2b87d8f42123c5: Bootstrap starting.
I20260928 21:58:31.309685 25203 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c52ef9dde5b84823ba2b87d8f42123c5: Neither blocks nor log segments found. Creating new log.
I20260928 21:58:31.309994 25203 log.cc:824] T 00000000000000000000000000000000 P c52ef9dde5b84823ba2b87d8f42123c5: Log is configured to *not* fsync() on all Append() calls
I20260928 21:58:31.310806 25203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c52ef9dde5b84823ba2b87d8f42123c5: No bootstrap required, opened a new log
Sep 28 21:58:31 dist-test-slave-k6lz krb5kdc[24972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632710, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.182.125@KRBTEST.COM
I20260928 21:58:31.312358 25203 raft_consensus.cc:348] T 00000000000000000000000000000000 P c52ef9dde5b84823ba2b87d8f42123c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5ae094ccd084b18bbf7523adb98e95c" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 45245 } } peers { permanent_uuid: "67a951337146439ba05102c122982ca2" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 46575 } } peers { permanent_uuid: "c52ef9dde5b84823ba2b87d8f42123c5" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 43045 } }
I20260928 21:58:31.312485 25203 raft_consensus.cc:729] T 00000000000000000000000000000000 P c52ef9dde5b84823ba2b87d8f42123c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c52ef9dde5b84823ba2b87d8f42123c5, State: Initialized, Role: FOLLOWER
I20260928 21:58:31.312563 25203 consensus_queue.cc:260] T 00000000000000000000000000000000 P c52ef9dde5b84823ba2b87d8f42123c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5ae094ccd084b18bbf7523adb98e95c" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 45245 } } peers { permanent_uuid: "67a951337146439ba05102c122982ca2" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 46575 } } peers { permanent_uuid: "c52ef9dde5b84823ba2b87d8f42123c5" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 43045 } }
I20260928 21:58:31.312875 25203 sys_catalog.cc:564] T 00000000000000000000000000000000 P c52ef9dde5b84823ba2b87d8f42123c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:58:31.312827 25209 sys_catalog.cc:455] T 00000000000000000000000000000000 P c52ef9dde5b84823ba2b87d8f42123c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5ae094ccd084b18bbf7523adb98e95c" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 45245 } } peers { permanent_uuid: "67a951337146439ba05102c122982ca2" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 46575 } } peers { permanent_uuid: "c52ef9dde5b84823ba2b87d8f42123c5" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 43045 } } }
I20260928 21:58:31.312965 25209 sys_catalog.cc:458] T 00000000000000000000000000000000 P c52ef9dde5b84823ba2b87d8f42123c5 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:58:31.314039 25132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:31.306738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42594 (local address 127.2.182.125:46575)
0928 21:58:31.306777 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:31.306779 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:31.306787 (+ 8us) server_negotiation.cc:407] Connection header received
0928 21:58:31.306810 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:31.306811 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:31.306821 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:31.306859 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:31.307410 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:31.307976 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:31.308865 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:31.309140 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:31.309201 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:31.311644 (+ 2443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:31.311647 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:31.311647 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:31.311655 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:31.313058 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:31.313365 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:31.313366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:31.313367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:31.313423 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:31.313580 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:31.313581 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:31.313581 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:31.313697 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:31.313843 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:31.313878 (+ 35us) server_negotiation.cc:299] Negotiation successful
0928 21:58:31.313950 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 21:58:31.314474 25053 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.182.124" port: 43045 } attempt: 2
W20260928 21:58:31.314488 25220 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c52ef9dde5b84823ba2b87d8f42123c5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:58:31.315352 25220 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:58:31.316011 2777 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:58:31.315471 25135 master_runner.cc:186] Error getting master registration for 127.2.182.126:45245: OK, instance_id { permanent_uuid: "d5ae094ccd084b18bbf7523adb98e95c" instance_seqno: 1790632710965556 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 21:58:31.317719 25135 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 21:58:31.317265 25027 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "d5ae094ccd084b18bbf7523adb98e95c"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59326
Sep 28 21:58:31 dist-test-slave-k6lz krb5kdc[24972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632710, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.126@KRBTEST.COM for kudu/127.2.182.124@KRBTEST.COM
I20260928 21:58:31.322800 25055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:31.314750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.124:43045 (local address 127.0.0.1:57930)
0928 21:58:31.314898 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:31.314910 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:31.314956 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:31.315315 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:31.315317 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:31.315623 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:58:31.315677 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:31.315681 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:31.316678 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:31.316679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:31.317445 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:31.317449 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:31.317944 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:31.317945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:31.318013 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:31.318525 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:58:31.318530 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:58:31.320004 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:58:31.321715 (+ 1711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:31.321716 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:31.321717 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:31.321930 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:31.322142 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:31.322142 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:31.322142 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:31.322199 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:31.322503 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:58:31.322504 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:58:31.322551 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 21:58:31.322608 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 21:58:31.322698 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130}
I20260928 21:58:31.322819 25206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:31.314807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57930 (local address 127.2.182.124:43045)
0928 21:58:31.314842 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:31.314844 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:31.314950 (+ 106us) server_negotiation.cc:407] Connection header received
0928 21:58:31.315001 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:31.315002 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:31.315011 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:31.315761 (+ 750us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:31.315810 (+ 49us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:31.316561 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:31.317556 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:31.317859 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:31.317931 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:31.320092 (+ 2161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:31.320093 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:31.320094 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:31.320120 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:31.321636 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:31.321997 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:31.321998 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:31.321999 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:31.322076 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:31.322252 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:31.322253 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:31.322253 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:31.322434 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:31.322501 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:31.322647 (+ 146us) server_negotiation.cc:299] Negotiation successful
0928 21:58:31.322728 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 21:58:31.324635 25053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5ae094ccd084b18bbf7523adb98e95c: Bootstrap starting.
I20260928 21:58:31.325426 25053 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5ae094ccd084b18bbf7523adb98e95c: Neither blocks nor log segments found. Creating new log.
I20260928 21:58:31.325730 25053 log.cc:824] T 00000000000000000000000000000000 P d5ae094ccd084b18bbf7523adb98e95c: Log is configured to *not* fsync() on all Append() calls
I20260928 21:58:31.326543 25053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5ae094ccd084b18bbf7523adb98e95c: No bootstrap required, opened a new log
I20260928 21:58:31.327972 25053 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5ae094ccd084b18bbf7523adb98e95c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5ae094ccd084b18bbf7523adb98e95c" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 45245 } } peers { permanent_uuid: "67a951337146439ba05102c122982ca2" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 46575 } } peers { permanent_uuid: "c52ef9dde5b84823ba2b87d8f42123c5" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 43045 } }
I20260928 21:58:31.328130 25053 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5ae094ccd084b18bbf7523adb98e95c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5ae094ccd084b18bbf7523adb98e95c, State: Initialized, Role: FOLLOWER
I20260928 21:58:31.328213 25053 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5ae094ccd084b18bbf7523adb98e95c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5ae094ccd084b18bbf7523adb98e95c" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 45245 } } peers { permanent_uuid: "67a951337146439ba05102c122982ca2" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 46575 } } peers { permanent_uuid: "c52ef9dde5b84823ba2b87d8f42123c5" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 43045 } }
I20260928 21:58:31.328465 25223 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5ae094ccd084b18bbf7523adb98e95c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5ae094ccd084b18bbf7523adb98e95c" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 45245 } } peers { permanent_uuid: "67a951337146439ba05102c122982ca2" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 46575 } } peers { permanent_uuid: "c52ef9dde5b84823ba2b87d8f42123c5" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 43045 } } }
I20260928 21:58:31.328511 25053 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5ae094ccd084b18bbf7523adb98e95c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:58:31.328527 25027 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5ae094ccd084b18bbf7523adb98e95c [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 21:58:31.328593 25223 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5ae094ccd084b18bbf7523adb98e95c [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:58:31.328657 25027 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5ae094ccd084b18bbf7523adb98e95c [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:58:31.329519 25027 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5ae094ccd084b18bbf7523adb98e95c [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5ae094ccd084b18bbf7523adb98e95c" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 45245 } } peers { permanent_uuid: "67a951337146439ba05102c122982ca2" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 46575 } } peers { permanent_uuid: "c52ef9dde5b84823ba2b87d8f42123c5" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 43045 } }
I20260928 21:58:31.329968 25027 leader_election.cc:288] T 00000000000000000000000000000000 P d5ae094ccd084b18bbf7523adb98e95c [CANDIDATE]: Term 1 election: Requested vote from peers 67a951337146439ba05102c122982ca2 (127.2.182.125:46575), c52ef9dde5b84823ba2b87d8f42123c5 (127.2.182.124:43045)
I20260928 21:58:31.330256 25177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d5ae094ccd084b18bbf7523adb98e95c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c52ef9dde5b84823ba2b87d8f42123c5"
I20260928 21:58:31.330387 25177 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c52ef9dde5b84823ba2b87d8f42123c5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:58:31.331385 25177 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c52ef9dde5b84823ba2b87d8f42123c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5ae094ccd084b18bbf7523adb98e95c in term 1.
I20260928 21:58:31.331621 25001 leader_election.cc:302] T 00000000000000000000000000000000 P d5ae094ccd084b18bbf7523adb98e95c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c52ef9dde5b84823ba2b87d8f42123c5, d5ae094ccd084b18bbf7523adb98e95c; no voters:
I20260928 21:58:31.331748 25223 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5ae094ccd084b18bbf7523adb98e95c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:58:31.331888 25223 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5ae094ccd084b18bbf7523adb98e95c [term 1 LEADER]: Becoming Leader. State: Replica: d5ae094ccd084b18bbf7523adb98e95c, State: Running, Role: LEADER
I20260928 21:58:31.331998 25223 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5ae094ccd084b18bbf7523adb98e95c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5ae094ccd084b18bbf7523adb98e95c" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 45245 } } peers { permanent_uuid: "67a951337146439ba05102c122982ca2" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 46575 } } peers { permanent_uuid: "c52ef9dde5b84823ba2b87d8f42123c5" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 43045 } }
W20260928 21:58:31.332646 25234 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d5ae094ccd084b18bbf7523adb98e95c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:58:31.332700 25234 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:58:31.332770 25241 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5ae094ccd084b18bbf7523adb98e95c [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5ae094ccd084b18bbf7523adb98e95c. Latest consensus state: current_term: 1 leader_uuid: "d5ae094ccd084b18bbf7523adb98e95c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5ae094ccd084b18bbf7523adb98e95c" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 45245 } } peers { permanent_uuid: "67a951337146439ba05102c122982ca2" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 46575 } } peers { permanent_uuid: "c52ef9dde5b84823ba2b87d8f42123c5" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 43045 } } }
I20260928 21:58:31.332854 25241 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5ae094ccd084b18bbf7523adb98e95c [sys.catalog]: This master's current role is: LEADER
I20260928 21:58:31.331787 24985 master_runner.cc:186] Error getting master registration for 127.2.182.125:46575: OK, instance_id { permanent_uuid: "67a951337146439ba05102c122982ca2" instance_seqno: 1790632711122525 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 21:58:31.334127 24985 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 21:58:31.339455 25243 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:58:31.339690 25243 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:58:31.340742 25056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:31.331500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59328 (local address 127.2.182.126:45245)
0928 21:58:31.331538 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:31.331540 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:31.333531 (+ 1991us) server_negotiation.cc:407] Connection header received
0928 21:58:31.333557 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:31.333558 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:31.333568 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:31.333613 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:31.334175 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:31.334701 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:31.335496 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:31.335756 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:31.335797 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:31.337340 (+ 1543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:31.337342 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:31.337342 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:31.337349 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:31.338946 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:31.339389 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:31.339390 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:31.339390 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:31.339449 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:31.339727 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:31.339728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:31.339728 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:31.339864 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:31.339949 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:31.340476 (+ 527us) server_negotiation.cc:299] Negotiation successful
0928 21:58:31.340654 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 21:58:31.341653 25177 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c52ef9dde5b84823ba2b87d8f42123c5 [term 1 FOLLOWER]: Refusing update from remote peer d5ae094ccd084b18bbf7523adb98e95c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:58:31.342017 25241 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d5ae094ccd084b18bbf7523adb98e95c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c52ef9dde5b84823ba2b87d8f42123c5" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 43045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:58:31.343590 25209 sys_catalog.cc:455] T 00000000000000000000000000000000 P c52ef9dde5b84823ba2b87d8f42123c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5ae094ccd084b18bbf7523adb98e95c. Latest consensus state: current_term: 1 leader_uuid: "d5ae094ccd084b18bbf7523adb98e95c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5ae094ccd084b18bbf7523adb98e95c" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 45245 } } peers { permanent_uuid: "67a951337146439ba05102c122982ca2" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 46575 } } peers { permanent_uuid: "c52ef9dde5b84823ba2b87d8f42123c5" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 43045 } } }
I20260928 21:58:31.343700 25209 sys_catalog.cc:458] T 00000000000000000000000000000000 P c52ef9dde5b84823ba2b87d8f42123c5 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:58:31.350772 25132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:31.333211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42602 (local address 127.2.182.125:46575)
0928 21:58:31.333265 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:31.333267 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:31.341487 (+ 8220us) server_negotiation.cc:407] Connection header received
0928 21:58:31.341608 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:31.341609 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:31.341622 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:31.341652 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:31.342505 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:31.343263 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:31.344567 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:31.345527 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:31.345587 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:31.347470 (+ 1883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:31.347472 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:31.347473 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:31.347483 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:31.349149 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:31.349583 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:31.349584 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:31.349585 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:31.349645 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:31.349835 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:31.349836 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:31.349836 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:31.349981 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:31.350328 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:31.350571 (+ 243us) server_negotiation.cc:299] Negotiation successful
0928 21:58:31.350668 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 21:58:31.353593 25223 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5ae094ccd084b18bbf7523adb98e95c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d5ae094ccd084b18bbf7523adb98e95c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5ae094ccd084b18bbf7523adb98e95c" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 45245 } } peers { permanent_uuid: "67a951337146439ba05102c122982ca2" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 46575 } } peers { permanent_uuid: "c52ef9dde5b84823ba2b87d8f42123c5" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 43045 } } }
I20260928 21:58:31.353731 25223 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5ae094ccd084b18bbf7523adb98e95c [sys.catalog]: This master's current role is: LEADER
I20260928 21:58:31.353832 25209 sys_catalog.cc:455] T 00000000000000000000000000000000 P c52ef9dde5b84823ba2b87d8f42123c5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d5ae094ccd084b18bbf7523adb98e95c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5ae094ccd084b18bbf7523adb98e95c" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 45245 } } peers { permanent_uuid: "67a951337146439ba05102c122982ca2" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 46575 } } peers { permanent_uuid: "c52ef9dde5b84823ba2b87d8f42123c5" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 43045 } } }
I20260928 21:58:31.353933 25209 sys_catalog.cc:458] T 00000000000000000000000000000000 P c52ef9dde5b84823ba2b87d8f42123c5 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:58:31.354146 25247 mvcc.cc:205] Tried to move back new op lower bound from 7334431585652056064 to 7334431585616445440. Current Snapshot: MvccSnapshot[applied={T|T < 7334431585652056064}]
I20260928 21:58:31.354401 25246 mvcc.cc:205] Tried to move back new op lower bound from 7334431585652056064 to 7334431585616445440. Current Snapshot: MvccSnapshot[applied={T|T < 7334431585652056064}]
I20260928 21:58:31.355394 25206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:31.333343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57946 (local address 127.2.182.124:43045)
0928 21:58:31.333381 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:31.333383 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:31.342389 (+ 9006us) server_negotiation.cc:407] Connection header received
0928 21:58:31.342474 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:31.342474 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:31.342484 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:31.342511 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:31.347418 (+ 4907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:31.348188 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:31.349241 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:31.349510 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:31.349572 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:31.350952 (+ 1380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:31.350953 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:31.350954 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:31.350962 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:31.352534 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:31.352917 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:31.352918 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:31.352918 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:31.353001 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:31.354298 (+ 1297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:31.354315 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:31.354315 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:31.354486 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:31.354636 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:31.355180 (+ 544us) server_negotiation.cc:299] Negotiation successful
0928 21:58:31.355270 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 21:58:31.355846 25243 catalog_manager.cc:1347] Generated new cluster ID: 3594254716f844de9173140968308352
I20260928 21:58:31.355897 25243 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:58:31.356606 25128 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.182.124" port: 43045 } attempt: 3
Sep 28 21:58:31 dist-test-slave-k6lz krb5kdc[24972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632711, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.125@KRBTEST.COM for kudu/127.2.182.124@KRBTEST.COM
I20260928 21:58:31.365949 25130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:31.356852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.124:43045 (local address 127.0.0.1:57950)
0928 21:58:31.356909 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:31.356921 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:31.356967 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:31.357093 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:31.357094 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:31.357279 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:58:31.357335 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:31.357337 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:31.358502 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:31.358503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:31.359394 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:31.359397 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:31.359868 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:31.359869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:31.359927 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:31.360399 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:58:31.360404 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:58:31.361845 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:58:31.364716 (+ 2871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:31.364717 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:31.364717 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:31.364931 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:31.365199 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:31.365199 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:31.365200 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:31.365254 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:31.365631 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:58:31.365632 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:58:31.365679 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 21:58:31.365732 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 21:58:31.365838 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260928 21:58:31.366371 25206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:31.356938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57950 (local address 127.2.182.124:43045)
0928 21:58:31.356969 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:31.356970 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:31.356978 (+ 8us) server_negotiation.cc:407] Connection header received
0928 21:58:31.357019 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:31.357020 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:31.357029 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:31.357075 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:31.357720 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:31.358431 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:31.359476 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:31.359810 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:31.359864 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:31.361947 (+ 2083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:31.361948 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:31.361949 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:31.361956 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:31.364587 (+ 2631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:31.364985 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:31.364986 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:31.364986 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:31.365105 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:31.365317 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:31.365318 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:31.365318 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:31.365538 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:31.365656 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:31.365795 (+ 139us) server_negotiation.cc:299] Negotiation successful
0928 21:58:31.365935 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 21:58:31.369043 25128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67a951337146439ba05102c122982ca2: Bootstrap starting.
I20260928 21:58:31.369973 25128 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67a951337146439ba05102c122982ca2: Neither blocks nor log segments found. Creating new log.
I20260928 21:58:31.370282 25128 log.cc:824] T 00000000000000000000000000000000 P 67a951337146439ba05102c122982ca2: Log is configured to *not* fsync() on all Append() calls
I20260928 21:58:31.371834 25128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67a951337146439ba05102c122982ca2: No bootstrap required, opened a new log
I20260928 21:58:31.373955 25128 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67a951337146439ba05102c122982ca2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5ae094ccd084b18bbf7523adb98e95c" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 45245 } } peers { permanent_uuid: "67a951337146439ba05102c122982ca2" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 46575 } } peers { permanent_uuid: "c52ef9dde5b84823ba2b87d8f42123c5" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 43045 } }
I20260928 21:58:31.374121 25128 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67a951337146439ba05102c122982ca2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67a951337146439ba05102c122982ca2, State: Initialized, Role: FOLLOWER
I20260928 21:58:31.374210 25128 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67a951337146439ba05102c122982ca2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5ae094ccd084b18bbf7523adb98e95c" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 45245 } } peers { permanent_uuid: "67a951337146439ba05102c122982ca2" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 46575 } } peers { permanent_uuid: "c52ef9dde5b84823ba2b87d8f42123c5" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 43045 } }
I20260928 21:58:31.374576 25128 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67a951337146439ba05102c122982ca2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:58:31.374679 25101 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67a951337146439ba05102c122982ca2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:58:31.375715 25101 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 67a951337146439ba05102c122982ca2 [term 1 FOLLOWER]: Refusing update from remote peer d5ae094ccd084b18bbf7523adb98e95c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:58:31.375910 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d5ae094ccd084b18bbf7523adb98e95c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "67a951337146439ba05102c122982ca2"
I20260928 21:58:31.376183 25241 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d5ae094ccd084b18bbf7523adb98e95c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67a951337146439ba05102c122982ca2" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 46575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:58:31.377449 25102 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 67a951337146439ba05102c122982ca2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5ae094ccd084b18bbf7523adb98e95c in term 1.
W20260928 21:58:31.378039 25263 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 67a951337146439ba05102c122982ca2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:58:31.378098 25263 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:58:31.374583 25252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67a951337146439ba05102c122982ca2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5ae094ccd084b18bbf7523adb98e95c" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 45245 } } peers { permanent_uuid: "67a951337146439ba05102c122982ca2" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 46575 } } peers { permanent_uuid: "c52ef9dde5b84823ba2b87d8f42123c5" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 43045 } } }
I20260928 21:58:31.380270 25252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67a951337146439ba05102c122982ca2 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:58:31.380338 25252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67a951337146439ba05102c122982ca2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d5ae094ccd084b18bbf7523adb98e95c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5ae094ccd084b18bbf7523adb98e95c" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 45245 } } peers { permanent_uuid: "67a951337146439ba05102c122982ca2" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 46575 } } peers { permanent_uuid: "c52ef9dde5b84823ba2b87d8f42123c5" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 43045 } } }
I20260928 21:58:31.380402 25252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67a951337146439ba05102c122982ca2 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:58:31.380609 25268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67a951337146439ba05102c122982ca2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5ae094ccd084b18bbf7523adb98e95c. Latest consensus state: current_term: 1 leader_uuid: "d5ae094ccd084b18bbf7523adb98e95c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5ae094ccd084b18bbf7523adb98e95c" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 45245 } } peers { permanent_uuid: "67a951337146439ba05102c122982ca2" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 46575 } } peers { permanent_uuid: "c52ef9dde5b84823ba2b87d8f42123c5" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 43045 } } }
I20260928 21:58:31.380650 25268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67a951337146439ba05102c122982ca2 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:58:31.403496 25241 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5ae094ccd084b18bbf7523adb98e95c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d5ae094ccd084b18bbf7523adb98e95c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5ae094ccd084b18bbf7523adb98e95c" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 45245 } } peers { permanent_uuid: "67a951337146439ba05102c122982ca2" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 46575 } } peers { permanent_uuid: "c52ef9dde5b84823ba2b87d8f42123c5" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 43045 } } }
I20260928 21:58:31.403659 25241 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5ae094ccd084b18bbf7523adb98e95c [sys.catalog]: This master's current role is: LEADER
I20260928 21:58:31.407253 25243 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:58:31.408100 25243 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:58:31.427757 25243 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d5ae094ccd084b18bbf7523adb98e95c: Generated new TSK 0
I20260928 21:58:31.427986 25243 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 21:58:31.453405 25056 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:58:31.451669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59330 (local address 127.2.182.126:45245)
0928 21:58:31.451718 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:31.451721 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:31.452706 (+ 985us) server_negotiation.cc:407] Connection header received
0928 21:58:31.452820 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:31.452822 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:31.452833 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:31.452863 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:31.453328 (+ 465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59330: BlockingRecv error: recv got EOF from 127.0.0.1:59330 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260928 21:58:31.454214 25132 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:58:31.452115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42608 (local address 127.2.182.125:46575)
0928 21:58:31.452157 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:31.452159 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:31.453678 (+ 1519us) server_negotiation.cc:407] Connection header received
0928 21:58:31.453786 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:31.453787 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:31.453798 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:31.453826 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:31.454152 (+ 326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42608: BlockingRecv error: recv got EOF from 127.0.0.1:42608 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 21:58:31.454970 25206 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:58:31.452459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57952 (local address 127.2.182.124:43045)
0928 21:58:31.452521 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:31.452524 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:31.454448 (+ 1924us) server_negotiation.cc:407] Connection header received
0928 21:58:31.454576 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:31.454577 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:31.454588 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:31.454614 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:31.454911 (+ 297us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57952: BlockingRecv error: recv got EOF from 127.0.0.1:57952 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260928 21:58:31.457736 25132 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:58:31.456361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42622 (local address 127.2.182.125:46575)
0928 21:58:31.456410 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:31.456412 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:31.457035 (+ 623us) server_negotiation.cc:407] Connection header received
0928 21:58:31.457222 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:31.457224 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:31.457243 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:31.457271 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:31.457646 (+ 375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42622: BlockingRecv error: recv got EOF from 127.0.0.1:42622 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260928 21:58:31.461079 25132 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:58:31.460268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42638 (local address 127.2.182.125:46575)
0928 21:58:31.460302 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:31.460304 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:31.460522 (+ 218us) server_negotiation.cc:407] Connection header received
0928 21:58:31.460638 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:31.460639 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:31.460649 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:31.460677 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:31.461028 (+ 351us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42638: BlockingRecv error: recv got EOF from 127.0.0.1:42638 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 21:58:31.465334 25132 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:58:31.464511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42644 (local address 127.2.182.125:46575)
0928 21:58:31.464547 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:31.464549 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:31.464788 (+ 239us) server_negotiation.cc:407] Connection header received
0928 21:58:31.464889 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:31.464890 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:31.464901 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:31.464929 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:31.465280 (+ 351us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42644: BlockingRecv error: recv got EOF from 127.0.0.1:42644 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 21:58:31.471788 25132 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:58:31.470641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42656 (local address 127.2.182.125:46575)
0928 21:58:31.470685 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:31.470687 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:31.471087 (+ 400us) server_negotiation.cc:407] Connection header received
0928 21:58:31.471198 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:31.471199 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:31.471211 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:31.471241 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:31.471730 (+ 489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42656: BlockingRecv error: recv got EOF from 127.0.0.1:42656 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 21:58:31.482033 25132 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:58:31.481136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42664 (local address 127.2.182.125:46575)
0928 21:58:31.481180 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:31.481182 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:31.481441 (+ 259us) server_negotiation.cc:407] Connection header received
0928 21:58:31.481541 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:31.481542 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:31.481553 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:31.481592 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:31.481973 (+ 381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42664: BlockingRecv error: recv got EOF from 127.0.0.1:42664 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 21:58:31.500280 25132 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:58:31.499434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42680 (local address 127.2.182.125:46575)
0928 21:58:31.499479 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:31.499482 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:31.499733 (+ 251us) server_negotiation.cc:407] Connection header received
0928 21:58:31.499826 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:31.499827 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:31.499838 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:31.499866 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:31.500219 (+ 353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42680: BlockingRecv error: recv got EOF from 127.0.0.1:42680 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 21:58:31.534387 25132 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:58:31.533686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42682 (local address 127.2.182.125:46575)
0928 21:58:31.533745 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:31.533747 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:31.533804 (+ 57us) server_negotiation.cc:407] Connection header received
0928 21:58:31.533863 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:31.533864 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:31.533876 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:31.533955 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:31.534327 (+ 372us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42682: BlockingRecv error: recv got EOF from 127.0.0.1:42682 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260928 21:58:31.600401 25132 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:58:31.599756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42696 (local address 127.2.182.125:46575)
0928 21:58:31.599834 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:31.599837 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:31.599899 (+ 62us) server_negotiation.cc:407] Connection header received
0928 21:58:31.599948 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:31.599949 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:31.599960 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:31.600005 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:31.600337 (+ 332us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42696: BlockingRecv error: recv got EOF from 127.0.0.1:42696 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260928 21:58:31.730265 25132 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:58:31.729561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42712 (local address 127.2.182.125:46575)
0928 21:58:31.729634 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:31.729636 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:31.729693 (+ 57us) server_negotiation.cc:407] Connection header received
0928 21:58:31.729745 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:31.729745 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:31.729756 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:31.729804 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:31.730206 (+ 402us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42712: BlockingRecv error: recv got EOF from 127.0.0.1:42712 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260928 21:58:31.988129 25132 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:58:31.987467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42718 (local address 127.2.182.125:46575)
0928 21:58:31.987596 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:31.987598 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:31.987607 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:58:31.987632 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:31.987633 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:31.987643 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:31.987701 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:31.988068 (+ 367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42718: BlockingRecv error: recv got EOF from 127.0.0.1:42718 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
I20260928 21:58:32.315892 25220 catalog_manager.cc:1259] Loaded cluster ID: 3594254716f844de9173140968308352
I20260928 21:58:32.315975 25220 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c52ef9dde5b84823ba2b87d8f42123c5: loading cluster ID for follower catalog manager: success
I20260928 21:58:32.316799 25220 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c52ef9dde5b84823ba2b87d8f42123c5: acquiring CA information for follower catalog manager: success
I20260928 21:58:32.317083 25220 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c52ef9dde5b84823ba2b87d8f42123c5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 21:58:32.378506 25263 catalog_manager.cc:1259] Loaded cluster ID: 3594254716f844de9173140968308352
I20260928 21:58:32.378593 25263 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 67a951337146439ba05102c122982ca2: loading cluster ID for follower catalog manager: success
I20260928 21:58:32.379426 25263 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 67a951337146439ba05102c122982ca2: acquiring CA information for follower catalog manager: success
I20260928 21:58:32.379693 25263 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 67a951337146439ba05102c122982ca2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 21:58:32.504066 25335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:32.501347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42728 (local address 127.2.182.125:46575)
0928 21:58:32.501477 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:32.501479 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:32.501489 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:58:32.501535 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:32.501536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:32.501596 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:32.501666 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:58:32.501858 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:32.502361 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:32.503213 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:32.503446 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:32.503575 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:32.503740 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:58:32.503882 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:58:32.503939 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:32.503977 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 21:58:32.503999 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":70,"threads_started":1}
I20260928 21:58:32.508495 25341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:32.505945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57962 (local address 127.2.182.124:43045)
0928 21:58:32.506046 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:32.506048 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:32.506060 (+ 12us) server_negotiation.cc:407] Connection header received
0928 21:58:32.506141 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:32.506141 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:32.506177 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:32.506245 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:58:32.506324 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:32.506807 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:32.507609 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:32.507822 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:32.507880 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:32.508135 (+ 255us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:58:32.508246 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:58:32.508322 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:32.508404 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 21:58:32.508424 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78,"thread_start_us":45,"threads_started":1}
I20260928 21:58:32.512990 25335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:32.510049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42740 (local address 127.2.182.125:46575)
0928 21:58:32.510089 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:32.510091 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:32.510220 (+ 129us) server_negotiation.cc:407] Connection header received
0928 21:58:32.510330 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:32.510331 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:32.510344 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:32.510416 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:58:32.510598 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:32.511119 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:32.512123 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:32.512404 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:32.512487 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:32.512652 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:58:32.512686 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:58:32.512730 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:32.512883 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 21:58:32.512903 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 21:58:32.513127 25341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:32.510341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57974 (local address 127.2.182.124:43045)
0928 21:58:32.510476 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:32.510477 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:32.510484 (+ 7us) server_negotiation.cc:407] Connection header received
0928 21:58:32.510506 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:32.510507 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:32.510519 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:32.510561 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:58:32.510763 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:32.511494 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:32.512271 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:32.512583 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:32.512613 (+ 30us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:32.512794 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:58:32.512824 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:58:32.512849 (+ 25us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:32.512956 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 21:58:32.513047 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260928 21:58:32.513276 25348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:32.510074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59346 (local address 127.2.182.126:45245)
0928 21:58:32.510180 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:32.510182 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:32.510306 (+ 124us) server_negotiation.cc:407] Connection header received
0928 21:58:32.510458 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:32.510459 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:32.510502 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:32.510574 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:58:32.510677 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:32.511457 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:32.512338 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:32.512640 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:32.512699 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:32.512844 (+ 145us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:58:32.512956 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:58:32.513010 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:32.513175 (+ 165us) server_negotiation.cc:299] Negotiation successful
0928 21:58:32.513203 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":61,"threads_started":1}
I20260928 21:58:32.515134 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 24985
I20260928 21:58:32.520022 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 25060
I20260928 21:58:32.524596 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 25135
2026-09-28T21:58:32Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2767 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632440055392-2777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:58:32 dist-test-slave-k6lz krb5kdc[25352](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:58:32 dist-test-slave-k6lz krb5kdc[25352](info): set up 2 sockets
Sep 28 21:58:32 dist-test-slave-k6lz krb5kdc[25352](info): commencing operation
krb5kdc: starting...
W20260928 21:58:33.491492 2777 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632440055392-2777-0/krb5kdc/test-user.keytab.
Sep 28 21:58:33 dist-test-slave-k6lz krb5kdc[25352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632713, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:58:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:58:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab.
Entry for principal kudu/127.2.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab.
I20260928 21:58:33.611138 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632440055392-2777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632440055392-2777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632440055392-2777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:45719
--webserver_interface=127.2.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab
--principal=kudu/127.2.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:39387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.126:45719,127.2.182.125:42329,127.2.182.124:40307
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:58:33.696797 25365 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 21:58:33.697032 25365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:58:33.697058 25365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:58:33.697086 25365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:58:33.698346 25365 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:58:33.698391 25365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:58:33.698410 25365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:58:33.698426 25365 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:58:33.698442 25365 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:58:33.700187 25365 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:39387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632440055392-2777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632440055392-2777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.126:45719,127.2.182.125:42329,127.2.182.124:40307
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.126.keytab
--principal=kudu/127.2.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.126:45719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25365
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632440055392-2777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:58:33.700456 25365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:58:33.700678 25365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:58:33.705540 25373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:33.705504 25371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:33.705741 25370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:33.705839 25365 server_base.cc:1034] running on GCE node
I20260928 21:58:33.706105 25365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:58:33.706362 25365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:58:33.707593 25365 hybrid_clock.cc:648] HybridClock initialized: now 1790632713707559 us; error 47 us; skew 500 ppm
Sep 28 21:58:33 dist-test-slave-k6lz krb5kdc[25352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632713, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:58:33.710230 25365 init.cc:377] Logged in from keytab as kudu/127.2.182.126@KRBTEST.COM (short username kudu)
I20260928 21:58:33.710688 25365 webserver.cc:460] Webserver started at http://127.2.182.126:35399/ using document root <none> and password file <none>
I20260928 21:58:33.710872 25365 fs_manager.cc:359] Metadata directory not provided
I20260928 21:58:33.710911 25365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:58:33.711010 25365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:58:33.712100 25365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632440055392-2777-0/minicluster-data/master-0/data/instance:
uuid: "0cf520f637244acda95f8c6163b63137"
format_stamp: "Formatted at 2026-09-28 21:58:33 on dist-test-slave-k6lz"
server_key: "9d4625d2b8def347567df9f534c41f8c"
server_key_iv: "dab2f6c7efdea40e6a94ec86f7917fba"
server_key_version: "encryptionkey@0"
I20260928 21:58:33.712423 25365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632440055392-2777-0/minicluster-data/master-0/wal/instance:
uuid: "0cf520f637244acda95f8c6163b63137"
format_stamp: "Formatted at 2026-09-28 21:58:33 on dist-test-slave-k6lz"
server_key: "9d4625d2b8def347567df9f534c41f8c"
server_key_iv: "dab2f6c7efdea40e6a94ec86f7917fba"
server_key_version: "encryptionkey@0"
I20260928 21:58:33.714035 25365 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:58:33.714843 25380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:33.715026 25365 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:58:33.715109 25365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632440055392-2777-0/minicluster-data/master-0/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632440055392-2777-0/minicluster-data/master-0/wal
uuid: "0cf520f637244acda95f8c6163b63137"
format_stamp: "Formatted at 2026-09-28 21:58:33 on dist-test-slave-k6lz"
server_key: "9d4625d2b8def347567df9f534c41f8c"
server_key_iv: "dab2f6c7efdea40e6a94ec86f7917fba"
server_key_version: "encryptionkey@0"
I20260928 21:58:33.715183 25365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632440055392-2777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632440055392-2777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632440055392-2777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:58:33.728291 25365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:58:33.730551 25365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:58:33.730674 25365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:58:33.734622 25365 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.126:45719
I20260928 21:58:33.734738 25432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.126:45719 every 8 connection(s)
I20260928 21:58:33.735005 25365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632440055392-2777-0/minicluster-data/master-0/data/info.pb
I20260928 21:58:33.736163 25433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:58:33.736423 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 25365
I20260928 21:58:33.736598 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632440055392-2777-0/minicluster-data/master-0/wal/instance
I20260928 21:58:33.736791 2777 external_mini_cluster.cc:1428] Setting key b76c0ff892f4d96d7c57d3df1eee35a6
I20260928 21:58:33.738467 25433 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.182.126" port: 45719 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.182.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.125@KRBTEST.COM" created.
Sep 28 21:58:33 dist-test-slave-k6lz krb5kdc[25352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632713, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.126@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:58:33.751204 25436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:33.739387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:45719 (local address 127.0.0.1:49462)
0928 21:58:33.739592 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:33.739640 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:33.740219 (+ 579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:33.740480 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:33.740483 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:33.740760 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:58:33.740914 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:33.740917 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:33.741903 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:33.741904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:33.742585 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:33.742587 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:33.743166 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:33.743168 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:33.743233 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:33.743920 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:58:33.743929 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:58:33.747876 (+ 3947us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:58:33.749791 (+ 1915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:33.749798 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:33.749800 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:33.749989 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:33.750217 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:33.750218 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:33.750218 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:33.750294 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:33.750636 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:58:33.750641 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:58:33.750702 (+ 61us) client_negotiation.cc:769] Sending connection context
0928 21:58:33.750783 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 21:58:33.751012 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":83,"thread_start_us":93,"threads_started":1}
I20260928 21:58:33.751206 25437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:33.739429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49462 (local address 127.2.182.126:45719)
0928 21:58:33.739649 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:33.739653 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:33.739728 (+ 75us) server_negotiation.cc:407] Connection header received
0928 21:58:33.740376 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:33.740378 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:33.740395 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:33.740477 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:33.741009 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:33.741814 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:33.742708 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:33.743054 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:33.743177 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:33.747978 (+ 4801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:33.747987 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:33.747987 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:33.748003 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:33.749699 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:33.750078 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:33.750081 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:33.750082 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:33.750163 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:33.750348 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:33.750349 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:33.750350 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:33.750576 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:33.750646 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:33.750826 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 21:58:33.750939 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":146,"threads_started":1}
I20260928 21:58:33.752090 25433 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.182.125" port: 42329 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:58:33.752434 25436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:33.752329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.125:42329 (local address 127.0.0.1:47158)
0928 21:58:33.752369 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:33.752410 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.125:42329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260928 21:58:33.752568 25381 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.182.125:42329: connect: Connection refused (error 111)
I20260928 21:58:33.752971 25436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:33.752905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.125:42329 (local address 127.0.0.1:47168)
0928 21:58:33.752930 (+ 25us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:33.752953 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.125:42329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260928 21:58:33.753067 25433 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.182.125:42329: Network error: Client connection negotiation failed: client connection to 127.2.182.125:42329: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.125.keytab.
Entry for principal kudu/127.2.182.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.125.keytab.
I20260928 21:58:33.762936 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632440055392-2777-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632440055392-2777-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632440055392-2777-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632440055392-2777-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.125:42329
--webserver_interface=127.2.182.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.125.keytab
--principal=kudu/127.2.182.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:39387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.126:45719,127.2.182.125:42329,127.2.182.124:40307
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:58:33.802353 25433 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.182.125" port: 42329 } attempt: 1
I20260928 21:58:33.802920 25436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:33.802774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.125:42329 (local address 127.0.0.1:47170)
0928 21:58:33.802838 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:33.802887 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.125:42329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 21:58:33.803289 25436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:33.803206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.125:42329 (local address 127.0.0.1:47186)
0928 21:58:33.803244 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:33.803271 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.125:42329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260928 21:58:33.803448 25433 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.182.125:42329: Network error: Client connection negotiation failed: client connection to 127.2.182.125:42329: connect: Connection refused (error 111)
W20260928 21:58:33.845914 25440 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 21:58:33.846150 25440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:58:33.846168 25440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:58:33.846189 25440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:58:33.847473 25440 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:58:33.847514 25440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:58:33.847528 25440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:58:33.847548 25440 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:58:33.847558 25440 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:58:33.849162 25440 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:39387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632440055392-2777-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632440055392-2777-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.126:45719,127.2.182.125:42329,127.2.182.124:40307
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.125.keytab
--principal=kudu/127.2.182.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.125:42329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632440055392-2777-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.182.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632440055392-2777-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:58:33.849409 25440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:58:33.849624 25440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:58:33.854705 25445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:33.854699 25448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:33.854763 25440 server_base.cc:1034] running on GCE node
W20260928 21:58:33.854758 25446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:33.855142 25440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:58:33.855456 25440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:58:33.856637 25440 hybrid_clock.cc:648] HybridClock initialized: now 1790632713856621 us; error 32 us; skew 500 ppm
Sep 28 21:58:33 dist-test-slave-k6lz krb5kdc[25352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632713, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:58:33.859323 25440 init.cc:377] Logged in from keytab as kudu/127.2.182.125@KRBTEST.COM (short username kudu)
I20260928 21:58:33.859925 25440 webserver.cc:460] Webserver started at http://127.2.182.125:33385/ using document root <none> and password file <none>
I20260928 21:58:33.860157 25440 fs_manager.cc:359] Metadata directory not provided
I20260928 21:58:33.860209 25440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:58:33.860332 25440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:58:33.861425 25440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632440055392-2777-0/minicluster-data/master-1/data/instance:
uuid: "27a1e245c6214921b3169f0b3b16be22"
format_stamp: "Formatted at 2026-09-28 21:58:33 on dist-test-slave-k6lz"
server_key: "805f40b9495b9fac161a6825c023d329"
server_key_iv: "b0c67acaaee8a66d2dc5959831b6882a"
server_key_version: "encryptionkey@0"
I20260928 21:58:33.861756 25440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632440055392-2777-0/minicluster-data/master-1/wal/instance:
uuid: "27a1e245c6214921b3169f0b3b16be22"
format_stamp: "Formatted at 2026-09-28 21:58:33 on dist-test-slave-k6lz"
server_key: "805f40b9495b9fac161a6825c023d329"
server_key_iv: "b0c67acaaee8a66d2dc5959831b6882a"
server_key_version: "encryptionkey@0"
I20260928 21:58:33.863163 25440 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:58:33.864156 25455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:33.864315 25440 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:58:33.864414 25440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632440055392-2777-0/minicluster-data/master-1/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632440055392-2777-0/minicluster-data/master-1/wal
uuid: "27a1e245c6214921b3169f0b3b16be22"
format_stamp: "Formatted at 2026-09-28 21:58:33 on dist-test-slave-k6lz"
server_key: "805f40b9495b9fac161a6825c023d329"
server_key_iv: "b0c67acaaee8a66d2dc5959831b6882a"
server_key_version: "encryptionkey@0"
I20260928 21:58:33.864483 25440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632440055392-2777-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632440055392-2777-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632440055392-2777-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:58:33.871610 25433 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.182.125" port: 42329 } attempt: 2
I20260928 21:58:33.872045 25436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:33.871931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.125:42329 (local address 127.0.0.1:47198)
0928 21:58:33.871986 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:33.872018 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.125:42329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 21:58:33.872403 25436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:33.872324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.125:42329 (local address 127.0.0.1:47208)
0928 21:58:33.872353 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:33.872381 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.125:42329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260928 21:58:33.872520 25433 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.182.125:42329: Network error: Client connection negotiation failed: client connection to 127.2.182.125:42329: connect: Connection refused (error 111)
I20260928 21:58:33.873512 25440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:58:33.876066 25440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:58:33.876194 25440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:58:33.880014 25440 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.125:42329
I20260928 21:58:33.880141 25507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.125:42329 every 8 connection(s)
I20260928 21:58:33.880401 25440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632440055392-2777-0/minicluster-data/master-1/data/info.pb
I20260928 21:58:33.881265 25508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:58:33.883039 25508 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.182.126" port: 45719 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:58:33.888325 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 25440
I20260928 21:58:33.888420 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632440055392-2777-0/minicluster-data/master-1/wal/instance
I20260928 21:58:33.888607 2777 external_mini_cluster.cc:1428] Setting key aa756a936371b5863c30420fea09f903
Sep 28 21:58:33 dist-test-slave-k6lz krb5kdc[25352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632713, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.125@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:58:33.893219 25437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:33.883921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49464 (local address 127.2.182.126:45719)
0928 21:58:33.884031 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:33.884034 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:33.884231 (+ 197us) server_negotiation.cc:407] Connection header received
0928 21:58:33.884784 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:33.884785 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:33.884875 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:33.884948 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:33.885689 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:33.886177 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:33.887158 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:33.887406 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:33.887461 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:33.889883 (+ 2422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:33.889885 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:33.889885 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:33.889893 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:33.891446 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:33.891979 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:33.891981 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:33.891981 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:33.892059 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:33.892426 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:33.892427 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:33.892427 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:33.892551 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:33.892692 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:33.893064 (+ 372us) server_negotiation.cc:299] Negotiation successful
0928 21:58:33.893149 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 21:58:33.893496 25510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:33.883870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:45719 (local address 127.0.0.1:49464)
0928 21:58:33.884084 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:33.884123 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:33.884635 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:33.885098 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:33.885101 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:33.885360 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:58:33.885592 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:33.885597 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:33.886260 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:33.886260 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:33.887076 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:33.887078 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:33.887504 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:33.887505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:33.887625 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:33.888304 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:58:33.888314 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:58:33.889781 (+ 1467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:58:33.891532 (+ 1751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:33.891543 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:33.891546 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:33.891874 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:33.892194 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:33.892195 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:33.892195 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:33.892358 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:33.892702 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:58:33.892711 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:58:33.892914 (+ 203us) client_negotiation.cc:769] Sending connection context
0928 21:58:33.893040 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 21:58:33.893242 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":92,"threads_started":1}
I20260928 21:58:33.894253 25508 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.182.125" port: 42329 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.182.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.182.124@KRBTEST.COM" created.
Sep 28 21:58:33 dist-test-slave-k6lz krb5kdc[25352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632713, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.125@KRBTEST.COM for kudu/127.2.182.125@KRBTEST.COM
I20260928 21:58:33.902948 25510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:33.894549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.125:42329 (local address 127.0.0.1:47210)
0928 21:58:33.894766 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:33.894777 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:33.894827 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:33.894967 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:33.894967 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:33.895160 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:58:33.895218 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:33.895220 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:33.896136 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:33.896137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:33.896627 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:33.896629 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:33.897087 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:33.897088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:33.897151 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:33.897540 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:58:33.897545 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:58:33.900176 (+ 2631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:58:33.901966 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:33.901967 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:33.901967 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:33.902125 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:33.902299 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:33.902300 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:33.902300 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:33.902352 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:33.902618 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:58:33.902619 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:58:33.902658 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 21:58:33.902717 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 21:58:33.902831 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 21:58:33.903005 25512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:33.894624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47210 (local address 127.2.182.125:42329)
0928 21:58:33.894783 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:33.894788 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:33.894826 (+ 38us) server_negotiation.cc:407] Connection header received
0928 21:58:33.894878 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:33.894880 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:33.894893 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:33.894966 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:33.895285 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:33.896061 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:33.896690 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:33.897017 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:33.897075 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:33.900275 (+ 3200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:33.900289 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:33.900289 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:33.900297 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:33.901882 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:33.902190 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:33.902193 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:33.902194 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:33.902248 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:33.902398 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:33.902399 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:33.902399 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:33.902559 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:33.902629 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:33.902764 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 21:58:33.902884 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":133,"thread_start_us":94,"threads_started":1}
I20260928 21:58:33.903609 25508 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.182.124" port: 40307 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:58:33.903939 25510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:33.903831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.124:40307 (local address 127.0.0.1:49204)
0928 21:58:33.903872 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:33.903915 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.124:40307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260928 21:58:33.904052 25456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.182.124:40307: connect: Connection refused (error 111)
I20260928 21:58:33.904429 25510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:33.904346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.124:40307 (local address 127.0.0.1:49206)
0928 21:58:33.904382 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:33.904405 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.124:40307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260928 21:58:33.904526 25508 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.182.124:40307: Network error: Client connection negotiation failed: client connection to 127.2.182.124:40307: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.182.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.124.keytab.
Entry for principal kudu/127.2.182.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.124.keytab.
I20260928 21:58:33.916509 2777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZMcOxj/build/release/bin/kudu
/tmp/dist-test-taskZMcOxj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632440055392-2777-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632440055392-2777-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632440055392-2777-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632440055392-2777-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.124:40307
--webserver_interface=127.2.182.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.124.keytab
--principal=kudu/127.2.182.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.182.84:39387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.126:45719,127.2.182.125:42329,127.2.182.124:40307
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632440055392-2777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632440055392-2777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632440055392-2777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:58:33.953745 25508 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.182.124" port: 40307 } attempt: 1
I20260928 21:58:33.954286 25510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:33.954131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.124:40307 (local address 127.0.0.1:49214)
0928 21:58:33.954210 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:33.954255 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.124:40307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260928 21:58:33.954648 25510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:33.954558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.124:40307 (local address 127.0.0.1:49218)
0928 21:58:33.954603 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:33.954626 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.124:40307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 21:58:33.954766 25508 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.182.124:40307: Network error: Client connection negotiation failed: client connection to 127.2.182.124:40307: connect: Connection refused (error 111)
I20260928 21:58:33.963685 25433 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.182.125" port: 42329 } attempt: 3
Sep 28 21:58:33 dist-test-slave-k6lz krb5kdc[25352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632713, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.126@KRBTEST.COM for kudu/127.2.182.125@KRBTEST.COM
I20260928 21:58:33.971526 25436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:33.964218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.125:42329 (local address 127.0.0.1:47212)
0928 21:58:33.964287 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:33.964299 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:33.964369 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:33.964500 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:33.964501 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:33.964686 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:58:33.964760 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:33.964762 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:33.965424 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:33.965425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:33.966244 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:33.966247 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:33.966745 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:33.966746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:33.966824 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:33.967300 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:58:33.967306 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:58:33.968819 (+ 1513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:58:33.970346 (+ 1527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:33.970347 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:33.970347 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:33.970557 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:33.970797 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:33.970798 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:33.970798 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:33.970862 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:33.971172 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:58:33.971173 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:58:33.971227 (+ 54us) client_negotiation.cc:769] Sending connection context
0928 21:58:33.971294 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 21:58:33.971423 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 21:58:33.971561 25512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:33.964249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47212 (local address 127.2.182.125:42329)
0928 21:58:33.964296 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:33.964298 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:33.964377 (+ 79us) server_negotiation.cc:407] Connection header received
0928 21:58:33.964415 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:33.964416 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:33.964429 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:33.964486 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:33.964847 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:33.965352 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:33.966336 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:33.966666 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:33.966733 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:33.968922 (+ 2189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:33.968924 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:33.968925 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:33.968932 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:33.970255 (+ 1323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:33.970667 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:33.970668 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:33.970668 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:33.970729 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:33.970969 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:33.970970 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:33.970971 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:33.971103 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:33.971170 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:33.971315 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 21:58:33.971423 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 21:58:33.971925 25433 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.182.124" port: 40307 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:58:33.972271 25436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:33.972170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.124:40307 (local address 127.0.0.1:49226)
0928 21:58:33.972211 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:33.972246 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.124:40307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260928 21:58:33.972556 25436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:33.972475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.124:40307 (local address 127.0.0.1:49240)
0928 21:58:33.972513 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:33.972538 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.124:40307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 21:58:33.972642 25433 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.182.124:40307: Network error: Client connection negotiation failed: client connection to 127.2.182.124:40307: connect: Connection refused (error 111)
W20260928 21:58:33.999418 25515 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 21:58:33.999641 25515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:58:33.999658 25515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:58:33.999677 25515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:58:34.000847 25515 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:58:34.000888 25515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:58:34.000901 25515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:58:34.000913 25515 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:58:34.000923 25515 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:58:34.002457 25515 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.84:39387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632440055392-2777-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632440055392-2777-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.126:45719,127.2.182.125:42329,127.2.182.124:40307
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632440055392-2777-0/krb5kdc/kudu_127.2.182.124.keytab
--principal=kudu/127.2.182.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.124:40307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632440055392-2777-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.182.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632440055392-2777-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:58:34.002712 25515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:58:34.002954 25515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:58:34.003814 25433 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.182.124" port: 40307 } attempt: 1
I20260928 21:58:34.004254 25436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:34.004121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.124:40307 (local address 127.0.0.1:49246)
0928 21:58:34.004181 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:34.004220 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.124:40307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 21:58:34.004696 25436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:34.004597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.124:40307 (local address 127.0.0.1:49262)
0928 21:58:34.004638 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:34.004670 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.124:40307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260928 21:58:34.004812 25433 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.182.124:40307: Network error: Client connection negotiation failed: client connection to 127.2.182.124:40307: connect: Connection refused (error 111)
W20260928 21:58:34.008253 25521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:34.008248 25520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:58:34.008248 25523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:58:34.008603 25515 server_base.cc:1034] running on GCE node
I20260928 21:58:34.008829 25515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:58:34.009104 25515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:58:34.010303 25515 hybrid_clock.cc:648] HybridClock initialized: now 1790632714010283 us; error 29 us; skew 500 ppm
Sep 28 21:58:34 dist-test-slave-k6lz krb5kdc[25352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632714, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:58:34.012841 25515 init.cc:377] Logged in from keytab as kudu/127.2.182.124@KRBTEST.COM (short username kudu)
I20260928 21:58:34.013334 25515 webserver.cc:460] Webserver started at http://127.2.182.124:42113/ using document root <none> and password file <none>
I20260928 21:58:34.013559 25515 fs_manager.cc:359] Metadata directory not provided
I20260928 21:58:34.013605 25515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:58:34.013722 25515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:58:34.014755 25515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632440055392-2777-0/minicluster-data/master-2/data/instance:
uuid: "36a53a3cc7ca436d9469898ebb081364"
format_stamp: "Formatted at 2026-09-28 21:58:34 on dist-test-slave-k6lz"
server_key: "963ebbd85b484321a8752c47e84d62c8"
server_key_iv: "e9ae1c28f4507311bc030431a46f9705"
server_key_version: "encryptionkey@0"
I20260928 21:58:34.015128 25515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632440055392-2777-0/minicluster-data/master-2/wal/instance:
uuid: "36a53a3cc7ca436d9469898ebb081364"
format_stamp: "Formatted at 2026-09-28 21:58:34 on dist-test-slave-k6lz"
server_key: "963ebbd85b484321a8752c47e84d62c8"
server_key_iv: "e9ae1c28f4507311bc030431a46f9705"
server_key_version: "encryptionkey@0"
I20260928 21:58:34.016799 25515 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:58:34.017576 25530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:58:34.017750 25515 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:58:34.017838 25515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632440055392-2777-0/minicluster-data/master-2/data,/tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632440055392-2777-0/minicluster-data/master-2/wal
uuid: "36a53a3cc7ca436d9469898ebb081364"
format_stamp: "Formatted at 2026-09-28 21:58:34 on dist-test-slave-k6lz"
server_key: "963ebbd85b484321a8752c47e84d62c8"
server_key_iv: "e9ae1c28f4507311bc030431a46f9705"
server_key_version: "encryptionkey@0"
I20260928 21:58:34.017910 25515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632440055392-2777-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632440055392-2777-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632440055392-2777-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:58:34.022895 25508 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.182.124" port: 40307 } attempt: 2
I20260928 21:58:34.023458 25510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:34.023290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.124:40307 (local address 127.0.0.1:49276)
0928 21:58:34.023377 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:34.023410 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.124:40307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260928 21:58:34.023744 25510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:34.023674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.124:40307 (local address 127.0.0.1:49286)
0928 21:58:34.023703 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:34.023724 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.182.124:40307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260928 21:58:34.023811 25508 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.182.124:40307: Network error: Client connection negotiation failed: client connection to 127.2.182.124:40307: connect: Connection refused (error 111)
I20260928 21:58:34.033011 25515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:58:34.035529 25515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:58:34.035678 25515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:58:34.040005 25515 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.124:40307
I20260928 21:58:34.040050 25582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.182.124:40307 every 8 connection(s)
I20260928 21:58:34.040441 25515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632440055392-2777-0/minicluster-data/master-2/data/info.pb
I20260928 21:58:34.041460 2777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZMcOxj/build/release/bin/kudu as pid 25515
I20260928 21:58:34.041564 2777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZMcOxj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632440055392-2777-0/minicluster-data/master-2/wal/instance
I20260928 21:58:34.041630 25583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:58:34.041795 2777 external_mini_cluster.cc:1428] Setting key bc1491f27162690b825f066dc26748e2
I20260928 21:58:34.043887 25583 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.182.126" port: 45719 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 21:58:34 dist-test-slave-k6lz krb5kdc[25352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632714, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.124@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
Sep 28 21:58:34 dist-test-slave-k6lz krb5kdc[25352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632713, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.182.124@KRBTEST.COM
I20260928 21:58:34.052429 25437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:34.044647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49480 (local address 127.2.182.126:45719)
0928 21:58:34.044697 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:34.044698 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:34.044803 (+ 105us) server_negotiation.cc:407] Connection header received
0928 21:58:34.044892 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:34.044893 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:34.044902 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:34.044951 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:34.045566 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:34.046096 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:34.046853 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:34.047052 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:34.047116 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:34.049271 (+ 2155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:34.049273 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:34.049274 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:34.049282 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:34.050953 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:34.051456 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:34.051457 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:34.051458 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:34.051520 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:34.051747 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:34.051748 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:34.051748 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:34.051874 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:34.051935 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:34.052249 (+ 314us) server_negotiation.cc:299] Negotiation successful
0928 21:58:34.052336 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 21:58:34.052629 25587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:34.044581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.126:45719 (local address 127.0.0.1:49480)
0928 21:58:34.044703 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:34.044740 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:34.044822 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:34.044969 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:34.044972 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:34.045349 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:58:34.045456 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:34.045459 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:34.046181 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:34.046182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:34.046772 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:34.046774 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:34.047143 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:34.047144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:34.047197 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:34.047550 (+ 353us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:58:34.047559 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:58:34.049185 (+ 1626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:58:34.051044 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:34.051049 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:34.051051 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:34.051381 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:34.051585 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:34.051586 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:34.051586 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:34.051686 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:34.051933 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:58:34.051938 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:58:34.052091 (+ 153us) client_negotiation.cc:769] Sending connection context
0928 21:58:34.052213 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 21:58:34.052368 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":70,"threads_started":1}
I20260928 21:58:34.052800 25586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:34.042278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49300 (local address 127.2.182.124:40307)
0928 21:58:34.042453 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:34.042463 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:34.042488 (+ 25us) server_negotiation.cc:407] Connection header received
0928 21:58:34.043184 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:34.043193 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:34.043244 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:34.043408 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:34.044140 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:34.045387 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:34.046337 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:34.046753 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:34.046853 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:34.049693 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:34.049701 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:34.049702 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:34.049708 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:34.051456 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:34.051802 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:34.051804 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:34.051804 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:34.051848 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:34.052017 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:34.052018 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:34.052018 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:34.052163 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:34.052284 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:34.052371 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 21:58:34.052429 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":76,"threads_started":1}
I20260928 21:58:34.053486 25583 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.182.125" port: 42329 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 21:58:34 dist-test-slave-k6lz krb5kdc[25352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632714, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.124@KRBTEST.COM for kudu/127.2.182.125@KRBTEST.COM
Sep 28 21:58:34 dist-test-slave-k6lz krb5kdc[25352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632713, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.182.126@KRBTEST.COM
I20260928 21:58:34.060825 25587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:34.053716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.125:42329 (local address 127.0.0.1:47222)
0928 21:58:34.053761 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:34.053772 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:34.053815 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:34.053943 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:34.053943 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:34.054070 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:58:34.054125 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:34.054126 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:34.054741 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:34.054742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:34.055265 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:34.055267 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:34.055640 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:34.055641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:34.055704 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:34.056092 (+ 388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:58:34.056097 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:58:34.057617 (+ 1520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:58:34.059624 (+ 2007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:34.059625 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:34.059626 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:34.059854 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:34.060086 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:34.060087 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:34.060087 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:34.060155 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:34.060421 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:58:34.060423 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:58:34.060471 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 21:58:34.060621 (+ 150us) client_negotiation.cc:240] Negotiation successful
0928 21:58:34.060722 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 21:58:34.060885 25512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:34.053774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47222 (local address 127.2.182.125:42329)
0928 21:58:34.053833 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:34.053835 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:34.053854 (+ 19us) server_negotiation.cc:407] Connection header received
0928 21:58:34.053885 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:34.053886 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:34.053896 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:34.053941 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:34.054189 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:34.054673 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:34.055369 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:34.055580 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:34.055642 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:34.057726 (+ 2084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:34.057728 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:34.057728 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:34.057736 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:34.059528 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:34.059944 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:34.059945 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:34.059945 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:34.060016 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:34.060232 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:34.060233 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:34.060233 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:34.060360 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:34.060426 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:34.060693 (+ 267us) server_negotiation.cc:299] Negotiation successful
0928 21:58:34.060794 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 21:58:34.061379 25437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:34.053728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49492 (local address 127.2.182.126:45719)
0928 21:58:34.053822 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:34.053824 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:34.053831 (+ 7us) server_negotiation.cc:407] Connection header received
0928 21:58:34.053850 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:34.053851 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:34.053860 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:34.053903 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:34.054325 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:34.054796 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:34.055527 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:34.055729 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:34.055781 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:34.058218 (+ 2437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:34.058219 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:34.058219 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:34.058226 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:34.059877 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:34.060223 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:34.060224 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:34.060224 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:34.060277 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:34.060439 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:34.060439 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:34.060440 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:34.060562 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:34.060740 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:34.060809 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 21:58:34.060882 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 21:58:34.061456 25583 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.182.124" port: 40307 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 21:58:34 dist-test-slave-k6lz krb5kdc[25352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632714, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.124@KRBTEST.COM for kudu/127.2.182.124@KRBTEST.COM
Sep 28 21:58:34 dist-test-slave-k6lz krb5kdc[25352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632713, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.182.125@KRBTEST.COM
I20260928 21:58:34.068192 25587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:34.061798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.124:40307 (local address 127.0.0.1:49302)
0928 21:58:34.061838 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:34.061849 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:34.061884 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:34.061999 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:34.061999 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:34.062144 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:58:34.062184 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:34.062187 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:34.063010 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:34.063011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:34.063723 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:34.063725 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:34.064198 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:34.064199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:34.064240 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:34.064593 (+ 353us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:58:34.064597 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:58:34.065804 (+ 1207us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:58:34.067281 (+ 1477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:34.067282 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:34.067282 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:34.067483 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:34.067646 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:34.067646 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:34.067646 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:34.067696 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:34.067897 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:58:34.067898 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:58:34.067933 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 21:58:34.067999 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 21:58:34.068101 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260928 21:58:34.068518 25586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:34.061845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49302 (local address 127.2.182.124:40307)
0928 21:58:34.061891 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:34.061892 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:34.061902 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:58:34.061922 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:34.061922 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:34.061931 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:34.061993 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:34.062243 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:34.062936 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:34.063789 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:34.064117 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:34.064192 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:34.065865 (+ 1673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:34.065867 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:34.065867 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:34.065873 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:34.067225 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:34.067542 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:34.067543 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:34.067543 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:34.067592 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:34.067744 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:34.067745 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:34.067745 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:34.067848 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:34.067897 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:34.068032 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 21:58:34.068181 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":73,"server-negotiator.queue_time_us":29}
I20260928 21:58:34.069409 25512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:34.062145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47224 (local address 127.2.182.125:42329)
0928 21:58:34.062186 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:34.062188 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:34.062198 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:58:34.062218 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:34.062219 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:34.062227 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:34.062273 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:34.062789 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:34.063485 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:34.064308 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:34.064593 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:34.064659 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:34.066733 (+ 2074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:34.066734 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:34.066734 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:34.066740 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:34.068341 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:34.068680 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:34.068681 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:34.068682 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:34.068734 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:34.068949 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:34.068949 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:34.068950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:34.069072 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:34.069229 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:34.069267 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 21:58:34.069329 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 21:58:34.070099 25583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36a53a3cc7ca436d9469898ebb081364: Bootstrap starting.
I20260928 21:58:34.070770 2777 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:58:34.070969 25583 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36a53a3cc7ca436d9469898ebb081364: Neither blocks nor log segments found. Creating new log.
I20260928 21:58:34.071228 25583 log.cc:824] T 00000000000000000000000000000000 P 36a53a3cc7ca436d9469898ebb081364: Log is configured to *not* fsync() on all Append() calls
I20260928 21:58:34.072176 25583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36a53a3cc7ca436d9469898ebb081364: No bootstrap required, opened a new log
I20260928 21:58:34.071683 25407 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "0cf520f637244acda95f8c6163b63137"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49492
I20260928 21:58:34.074304 25583 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36a53a3cc7ca436d9469898ebb081364 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf520f637244acda95f8c6163b63137" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 45719 } } peers { permanent_uuid: "27a1e245c6214921b3169f0b3b16be22" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 42329 } } peers { permanent_uuid: "36a53a3cc7ca436d9469898ebb081364" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 40307 } }
I20260928 21:58:34.074496 25583 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36a53a3cc7ca436d9469898ebb081364 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36a53a3cc7ca436d9469898ebb081364, State: Initialized, Role: FOLLOWER
I20260928 21:58:34.074587 25583 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36a53a3cc7ca436d9469898ebb081364 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf520f637244acda95f8c6163b63137" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 45719 } } peers { permanent_uuid: "27a1e245c6214921b3169f0b3b16be22" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 42329 } } peers { permanent_uuid: "36a53a3cc7ca436d9469898ebb081364" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 40307 } }
I20260928 21:58:34.074939 25589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36a53a3cc7ca436d9469898ebb081364 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf520f637244acda95f8c6163b63137" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 45719 } } peers { permanent_uuid: "27a1e245c6214921b3169f0b3b16be22" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 42329 } } peers { permanent_uuid: "36a53a3cc7ca436d9469898ebb081364" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 40307 } } }
I20260928 21:58:34.074990 25583 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36a53a3cc7ca436d9469898ebb081364 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:58:34.075034 25589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36a53a3cc7ca436d9469898ebb081364 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 21:58:34.076886 25600 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 36a53a3cc7ca436d9469898ebb081364: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:58:34.076941 25600 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:58:34.077391 25515 master_runner.cc:186] Error getting master registration for 127.2.182.126:45719: OK, instance_id { permanent_uuid: "0cf520f637244acda95f8c6163b63137" instance_seqno: 1790632713733518 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 21:58:34.079013 25515 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 21:58:34.079947 25433 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.182.124" port: 40307 } attempt: 2
Sep 28 21:58:34 dist-test-slave-k6lz krb5kdc[25352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632713, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.126@KRBTEST.COM for kudu/127.2.182.124@KRBTEST.COM
I20260928 21:58:34.087884 25586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:34.080283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49304 (local address 127.2.182.124:40307)
0928 21:58:34.080327 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:34.080329 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:34.080337 (+ 8us) server_negotiation.cc:407] Connection header received
0928 21:58:34.080399 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:34.080399 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:34.080407 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:34.080450 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:34.080820 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:34.081597 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:34.082594 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:34.082893 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:34.082948 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:34.085165 (+ 2217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:34.085167 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:34.085167 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:34.085175 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:34.086699 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:34.087064 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:34.087065 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:34.087065 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:34.087123 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:34.087294 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:34.087294 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:34.087295 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:34.087509 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:34.087575 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:34.087713 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 21:58:34.087790 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 21:58:34.087886 25436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:34.080224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.124:40307 (local address 127.0.0.1:49304)
0928 21:58:34.080274 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:34.080287 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:34.080340 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:34.080483 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:34.080484 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:34.080682 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:58:34.080735 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:34.080738 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:34.081685 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:34.081686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:34.082514 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:34.082517 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:34.082958 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:34.082960 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:34.083022 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:34.083543 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:58:34.083548 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:58:34.085077 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:58:34.086779 (+ 1702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:34.086780 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:34.086781 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:34.086996 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:34.087186 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:34.087186 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:34.087187 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:34.087243 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:34.087567 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:58:34.087569 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:58:34.087618 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 21:58:34.087681 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 21:58:34.087781 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260928 21:58:34.089650 25433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cf520f637244acda95f8c6163b63137: Bootstrap starting.
I20260928 21:58:34.090370 25433 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0cf520f637244acda95f8c6163b63137: Neither blocks nor log segments found. Creating new log.
I20260928 21:58:34.090648 25433 log.cc:824] T 00000000000000000000000000000000 P 0cf520f637244acda95f8c6163b63137: Log is configured to *not* fsync() on all Append() calls
I20260928 21:58:34.091391 25433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cf520f637244acda95f8c6163b63137: No bootstrap required, opened a new log
I20260928 21:58:34.092731 25433 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0cf520f637244acda95f8c6163b63137 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf520f637244acda95f8c6163b63137" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 45719 } } peers { permanent_uuid: "27a1e245c6214921b3169f0b3b16be22" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 42329 } } peers { permanent_uuid: "36a53a3cc7ca436d9469898ebb081364" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 40307 } }
I20260928 21:58:34.092875 25433 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0cf520f637244acda95f8c6163b63137 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cf520f637244acda95f8c6163b63137, State: Initialized, Role: FOLLOWER
I20260928 21:58:34.092957 25433 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0cf520f637244acda95f8c6163b63137 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf520f637244acda95f8c6163b63137" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 45719 } } peers { permanent_uuid: "27a1e245c6214921b3169f0b3b16be22" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 42329 } } peers { permanent_uuid: "36a53a3cc7ca436d9469898ebb081364" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 40307 } }
I20260928 21:58:34.093284 25433 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0cf520f637244acda95f8c6163b63137 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:58:34.093257 25603 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cf520f637244acda95f8c6163b63137 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf520f637244acda95f8c6163b63137" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 45719 } } peers { permanent_uuid: "27a1e245c6214921b3169f0b3b16be22" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 42329 } } peers { permanent_uuid: "36a53a3cc7ca436d9469898ebb081364" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 40307 } } }
I20260928 21:58:34.093346 25603 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cf520f637244acda95f8c6163b63137 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:58:34.093377 25407 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0cf520f637244acda95f8c6163b63137 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 21:58:34.093425 25407 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0cf520f637244acda95f8c6163b63137 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:58:34.094245 25407 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0cf520f637244acda95f8c6163b63137 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf520f637244acda95f8c6163b63137" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 45719 } } peers { permanent_uuid: "27a1e245c6214921b3169f0b3b16be22" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 42329 } } peers { permanent_uuid: "36a53a3cc7ca436d9469898ebb081364" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 40307 } }
I20260928 21:58:34.094508 25407 leader_election.cc:288] T 00000000000000000000000000000000 P 0cf520f637244acda95f8c6163b63137 [CANDIDATE]: Term 1 election: Requested vote from peers 27a1e245c6214921b3169f0b3b16be22 (127.2.182.125:42329), 36a53a3cc7ca436d9469898ebb081364 (127.2.182.124:40307)
I20260928 21:58:34.094985 25557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0cf520f637244acda95f8c6163b63137" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "36a53a3cc7ca436d9469898ebb081364"
I20260928 21:58:34.095163 25557 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36a53a3cc7ca436d9469898ebb081364 [term 0 FOLLOWER]: Advancing to term 1
W20260928 21:58:34.095856 25614 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0cf520f637244acda95f8c6163b63137: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:58:34.096220 25614 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:58:34.096217 25557 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 36a53a3cc7ca436d9469898ebb081364 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cf520f637244acda95f8c6163b63137 in term 1.
I20260928 21:58:34.096439 25381 leader_election.cc:302] T 00000000000000000000000000000000 P 0cf520f637244acda95f8c6163b63137 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cf520f637244acda95f8c6163b63137, 36a53a3cc7ca436d9469898ebb081364; no voters:
I20260928 21:58:34.096549 25603 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0cf520f637244acda95f8c6163b63137 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:58:34.096696 25603 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0cf520f637244acda95f8c6163b63137 [term 1 LEADER]: Becoming Leader. State: Replica: 0cf520f637244acda95f8c6163b63137, State: Running, Role: LEADER
I20260928 21:58:34.096787 25603 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0cf520f637244acda95f8c6163b63137 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf520f637244acda95f8c6163b63137" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 45719 } } peers { permanent_uuid: "27a1e245c6214921b3169f0b3b16be22" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 42329 } } peers { permanent_uuid: "36a53a3cc7ca436d9469898ebb081364" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 40307 } }
I20260928 21:58:34.097338 25365 master_runner.cc:186] Error getting master registration for 127.2.182.125:42329: OK, instance_id { permanent_uuid: "27a1e245c6214921b3169f0b3b16be22" instance_seqno: 1790632713878910 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 21:58:34.099687 25365 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 21:58:34.100000 25620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cf520f637244acda95f8c6163b63137 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0cf520f637244acda95f8c6163b63137. Latest consensus state: current_term: 1 leader_uuid: "0cf520f637244acda95f8c6163b63137" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf520f637244acda95f8c6163b63137" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 45719 } } peers { permanent_uuid: "27a1e245c6214921b3169f0b3b16be22" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 42329 } } peers { permanent_uuid: "36a53a3cc7ca436d9469898ebb081364" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 40307 } } }
I20260928 21:58:34.100160 25620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cf520f637244acda95f8c6163b63137 [sys.catalog]: This master's current role is: LEADER
I20260928 21:58:34.104563 25437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:34.096847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49504 (local address 127.2.182.126:45719)
0928 21:58:34.097096 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:34.097098 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:34.097559 (+ 461us) server_negotiation.cc:407] Connection header received
0928 21:58:34.097646 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:34.097646 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:34.097657 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:34.097684 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:34.098433 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:34.099116 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:34.100139 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:34.100417 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:34.100462 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:34.101977 (+ 1515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:34.101979 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:34.101979 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:34.101986 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:34.103475 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:34.103820 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:34.103821 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:34.103821 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:34.103870 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:34.104039 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:34.104040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:34.104040 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:34.104180 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:34.104263 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:34.104410 (+ 147us) server_negotiation.cc:299] Negotiation successful
0928 21:58:34.104478 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226}
I20260928 21:58:34.111238 25512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:34.097223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47228 (local address 127.2.182.125:42329)
0928 21:58:34.097263 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:34.097264 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:34.104670 (+ 7406us) server_negotiation.cc:407] Connection header received
0928 21:58:34.104769 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:34.104770 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:34.104782 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:34.104813 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:34.105479 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:34.105957 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:34.106877 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:34.107096 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:34.107149 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:34.108441 (+ 1292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:34.108443 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:34.108443 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:34.108450 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:34.110039 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:34.110403 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:34.110404 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:34.110404 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:34.110471 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:34.110643 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:34.110644 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:34.110644 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:34.110790 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:34.110874 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:34.111061 (+ 187us) server_negotiation.cc:299] Negotiation successful
0928 21:58:34.111133 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 21:58:34.114921 25508 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.182.124" port: 40307 } attempt: 3
I20260928 21:58:34.116209 25625 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:58:34.116446 25625 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 21:58:34 dist-test-slave-k6lz krb5kdc[25352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632713, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.182.125@KRBTEST.COM for kudu/127.2.182.124@KRBTEST.COM
I20260928 21:58:34.123404 25586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:34.097341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49310 (local address 127.2.182.124:40307)
0928 21:58:34.097391 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:34.097394 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:34.108028 (+ 10634us) server_negotiation.cc:407] Connection header received
0928 21:58:34.108172 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:34.108174 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:34.108188 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:34.108238 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:34.116096 (+ 7858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:34.116723 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:34.118218 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:34.118531 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:34.118569 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:34.120088 (+ 1519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:34.120089 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:34.120089 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:34.120096 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:34.122057 (+ 1961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:34.122423 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:34.122424 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:34.122424 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:34.122478 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:34.122668 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:34.122668 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:34.122669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:34.122793 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:34.122873 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:34.123213 (+ 340us) server_negotiation.cc:299] Negotiation successful
0928 21:58:34.123288 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 21:58:34.124377 25510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:34.115219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.182.124:40307 (local address 127.0.0.1:49316)
0928 21:58:34.115268 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 21:58:34.115278 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:58:34.115315 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:58:34.115571 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:58:34.115572 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:58:34.115722 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:58:34.115761 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:34.115762 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:34.116784 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:34.116785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:34.117561 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:58:34.117563 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:58:34.118040 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:58:34.118041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:58:34.118090 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:34.118493 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:58:34.118497 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:58:34.119985 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:58:34.123075 (+ 3090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:34.123076 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:34.123077 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:34.123301 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:34.123647 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:58:34.123648 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:58:34.123649 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:58:34.123720 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:58:34.124077 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:58:34.124078 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:58:34.124132 (+ 54us) client_negotiation.cc:769] Sending connection context
0928 21:58:34.124187 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 21:58:34.124280 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 21:58:34.124871 25557 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 36a53a3cc7ca436d9469898ebb081364 [term 1 FOLLOWER]: Refusing update from remote peer 0cf520f637244acda95f8c6163b63137: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:58:34.124931 25626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:34.115264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49316 (local address 127.2.182.124:40307)
0928 21:58:34.115471 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:34.115473 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:34.115479 (+ 6us) server_negotiation.cc:407] Connection header received
0928 21:58:34.115500 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:34.115501 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:34.115510 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:34.115569 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:34.115831 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:34.116690 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:34.117650 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:34.117969 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:34.118029 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:34.120080 (+ 2051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:58:34.120081 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:58:34.120082 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:58:34.120088 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:58:34.122947 (+ 2859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:34.123442 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:34.123443 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:34.123443 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:34.123545 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:58:34.123796 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:58:34.123797 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:58:34.123797 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:58:34.123988 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:58:34.124559 (+ 571us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:34.124621 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 21:58:34.124747 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":124,"threads_started":1}
I20260928 21:58:34.125175 25620 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0cf520f637244acda95f8c6163b63137 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36a53a3cc7ca436d9469898ebb081364" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 40307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:58:34.126446 25508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27a1e245c6214921b3169f0b3b16be22: Bootstrap starting.
I20260928 21:58:34.126443 25589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36a53a3cc7ca436d9469898ebb081364 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0cf520f637244acda95f8c6163b63137. Latest consensus state: current_term: 1 leader_uuid: "0cf520f637244acda95f8c6163b63137" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf520f637244acda95f8c6163b63137" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 45719 } } peers { permanent_uuid: "27a1e245c6214921b3169f0b3b16be22" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 42329 } } peers { permanent_uuid: "36a53a3cc7ca436d9469898ebb081364" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 40307 } } }
I20260928 21:58:34.126524 25589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36a53a3cc7ca436d9469898ebb081364 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:58:34.127205 25508 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 27a1e245c6214921b3169f0b3b16be22: Neither blocks nor log segments found. Creating new log.
I20260928 21:58:34.127526 25508 log.cc:824] T 00000000000000000000000000000000 P 27a1e245c6214921b3169f0b3b16be22: Log is configured to *not* fsync() on all Append() calls
I20260928 21:58:34.132579 25620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cf520f637244acda95f8c6163b63137 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0cf520f637244acda95f8c6163b63137" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf520f637244acda95f8c6163b63137" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 45719 } } peers { permanent_uuid: "27a1e245c6214921b3169f0b3b16be22" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 42329 } } peers { permanent_uuid: "36a53a3cc7ca436d9469898ebb081364" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 40307 } } }
I20260928 21:58:34.132701 25620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cf520f637244acda95f8c6163b63137 [sys.catalog]: This master's current role is: LEADER
I20260928 21:58:34.132839 25508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27a1e245c6214921b3169f0b3b16be22: No bootstrap required, opened a new log
I20260928 21:58:34.133191 25589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36a53a3cc7ca436d9469898ebb081364 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0cf520f637244acda95f8c6163b63137" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf520f637244acda95f8c6163b63137" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 45719 } } peers { permanent_uuid: "27a1e245c6214921b3169f0b3b16be22" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 42329 } } peers { permanent_uuid: "36a53a3cc7ca436d9469898ebb081364" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 40307 } } }
I20260928 21:58:34.133277 25589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36a53a3cc7ca436d9469898ebb081364 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:58:34.133466 25628 mvcc.cc:205] Tried to move back new op lower bound from 7334431597052596224 to 7334431596941037568. Current Snapshot: MvccSnapshot[applied={T|T < 7334431597052596224}]
I20260928 21:58:34.134042 25627 mvcc.cc:205] Tried to move back new op lower bound from 7334431597052596224 to 7334431596941037568. Current Snapshot: MvccSnapshot[applied={T|T < 7334431597052596224}]
I20260928 21:58:34.134877 25508 raft_consensus.cc:348] T 00000000000000000000000000000000 P 27a1e245c6214921b3169f0b3b16be22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf520f637244acda95f8c6163b63137" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 45719 } } peers { permanent_uuid: "27a1e245c6214921b3169f0b3b16be22" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 42329 } } peers { permanent_uuid: "36a53a3cc7ca436d9469898ebb081364" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 40307 } }
I20260928 21:58:34.135016 25508 raft_consensus.cc:729] T 00000000000000000000000000000000 P 27a1e245c6214921b3169f0b3b16be22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27a1e245c6214921b3169f0b3b16be22, State: Initialized, Role: FOLLOWER
I20260928 21:58:34.135100 25508 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27a1e245c6214921b3169f0b3b16be22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf520f637244acda95f8c6163b63137" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 45719 } } peers { permanent_uuid: "27a1e245c6214921b3169f0b3b16be22" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 42329 } } peers { permanent_uuid: "36a53a3cc7ca436d9469898ebb081364" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 40307 } }
I20260928 21:58:34.135456 25625 catalog_manager.cc:1347] Generated new cluster ID: 58eecb7602d646dbb84b22c1d498ae0d
I20260928 21:58:34.135468 25508 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27a1e245c6214921b3169f0b3b16be22 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:58:34.135509 25625 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:58:34.136708 25482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0cf520f637244acda95f8c6163b63137" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "27a1e245c6214921b3169f0b3b16be22"
I20260928 21:58:34.136823 25482 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 27a1e245c6214921b3169f0b3b16be22 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:58:34.137676 25482 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 27a1e245c6214921b3169f0b3b16be22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cf520f637244acda95f8c6163b63137 in term 1.
W20260928 21:58:34.138728 25644 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 27a1e245c6214921b3169f0b3b16be22: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:58:34.138784 25644 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:58:34.139091 25633 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27a1e245c6214921b3169f0b3b16be22 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf520f637244acda95f8c6163b63137" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 45719 } } peers { permanent_uuid: "27a1e245c6214921b3169f0b3b16be22" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 42329 } } peers { permanent_uuid: "36a53a3cc7ca436d9469898ebb081364" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 40307 } } }
I20260928 21:58:34.139179 25633 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27a1e245c6214921b3169f0b3b16be22 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:58:34.139456 25481 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 27a1e245c6214921b3169f0b3b16be22 [term 1 FOLLOWER]: Refusing update from remote peer 0cf520f637244acda95f8c6163b63137: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:58:34.140179 25603 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0cf520f637244acda95f8c6163b63137 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27a1e245c6214921b3169f0b3b16be22" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 42329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:58:34.142282 25633 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27a1e245c6214921b3169f0b3b16be22 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0cf520f637244acda95f8c6163b63137. Latest consensus state: current_term: 1 leader_uuid: "0cf520f637244acda95f8c6163b63137" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf520f637244acda95f8c6163b63137" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 45719 } } peers { permanent_uuid: "27a1e245c6214921b3169f0b3b16be22" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 42329 } } peers { permanent_uuid: "36a53a3cc7ca436d9469898ebb081364" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 40307 } } }
I20260928 21:58:34.142369 25633 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27a1e245c6214921b3169f0b3b16be22 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:58:34.142481 25648 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27a1e245c6214921b3169f0b3b16be22 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0cf520f637244acda95f8c6163b63137" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf520f637244acda95f8c6163b63137" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 45719 } } peers { permanent_uuid: "27a1e245c6214921b3169f0b3b16be22" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 42329 } } peers { permanent_uuid: "36a53a3cc7ca436d9469898ebb081364" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 40307 } } }
I20260928 21:58:34.142578 25648 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27a1e245c6214921b3169f0b3b16be22 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:58:34.181301 25603 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cf520f637244acda95f8c6163b63137 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0cf520f637244acda95f8c6163b63137" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf520f637244acda95f8c6163b63137" member_type: VOTER last_known_addr { host: "127.2.182.126" port: 45719 } } peers { permanent_uuid: "27a1e245c6214921b3169f0b3b16be22" member_type: VOTER last_known_addr { host: "127.2.182.125" port: 42329 } } peers { permanent_uuid: "36a53a3cc7ca436d9469898ebb081364" member_type: VOTER last_known_addr { host: "127.2.182.124" port: 40307 } } }
I20260928 21:58:34.181439 25603 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cf520f637244acda95f8c6163b63137 [sys.catalog]: This master's current role is: LEADER
I20260928 21:58:34.184491 25625 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:58:34.185245 25625 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:58:34.191323 25625 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0cf520f637244acda95f8c6163b63137: Generated new TSK 0
I20260928 21:58:34.191550 25625 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 21:58:34.218802 25437 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:58:34.217062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49520 (local address 127.2.182.126:45719)
0928 21:58:34.217111 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:34.217113 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:34.218019 (+ 906us) server_negotiation.cc:407] Connection header received
0928 21:58:34.218152 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:34.218154 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:34.218166 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:34.218195 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:34.218727 (+ 532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49520: BlockingRecv error: recv got EOF from 127.0.0.1:49520 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260928 21:58:34.219585 25512 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:58:34.217478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47242 (local address 127.2.182.125:42329)
0928 21:58:34.217550 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:34.217553 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:34.219047 (+ 1494us) server_negotiation.cc:407] Connection header received
0928 21:58:34.219139 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:34.219140 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:34.219151 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:34.219178 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:34.219508 (+ 330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47242: BlockingRecv error: recv got EOF from 127.0.0.1:47242 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260928 21:58:34.220352 25626 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:58:34.217824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49326 (local address 127.2.182.124:40307)
0928 21:58:34.217884 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:34.217886 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:34.219840 (+ 1954us) server_negotiation.cc:407] Connection header received
0928 21:58:34.219930 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:34.219931 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:34.219957 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:34.219987 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:34.220289 (+ 302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49326: BlockingRecv error: recv got EOF from 127.0.0.1:49326 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260928 21:58:34.223012 25512 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:58:34.222101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47254 (local address 127.2.182.125:42329)
0928 21:58:34.222139 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:34.222141 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:34.222391 (+ 250us) server_negotiation.cc:407] Connection header received
0928 21:58:34.222522 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:34.222524 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:34.222557 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:34.222602 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:34.222960 (+ 358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47254: BlockingRecv error: recv got EOF from 127.0.0.1:47254 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 21:58:34.226323 25512 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:58:34.225526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47256 (local address 127.2.182.125:42329)
0928 21:58:34.225579 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:34.225581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:34.225806 (+ 225us) server_negotiation.cc:407] Connection header received
0928 21:58:34.225898 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:34.225899 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:34.225910 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:34.225938 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:34.226267 (+ 329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47256: BlockingRecv error: recv got EOF from 127.0.0.1:47256 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260928 21:58:34.230497 25512 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:58:34.229689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47264 (local address 127.2.182.125:42329)
0928 21:58:34.229728 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:34.229729 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:34.229950 (+ 221us) server_negotiation.cc:407] Connection header received
0928 21:58:34.230051 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:34.230052 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:34.230063 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:34.230092 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:34.230444 (+ 352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47264: BlockingRecv error: recv got EOF from 127.0.0.1:47264 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 21:58:34.236554 25512 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:58:34.235787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47276 (local address 127.2.182.125:42329)
0928 21:58:34.235825 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:34.235827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:34.236033 (+ 206us) server_negotiation.cc:407] Connection header received
0928 21:58:34.236123 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:34.236124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:34.236135 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:34.236161 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:34.236484 (+ 323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47276: BlockingRecv error: recv got EOF from 127.0.0.1:47276 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 21:58:34.246670 25512 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:58:34.245803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47278 (local address 127.2.182.125:42329)
0928 21:58:34.245845 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:34.245847 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:34.246105 (+ 258us) server_negotiation.cc:407] Connection header received
0928 21:58:34.246205 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:34.246206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:34.246218 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:34.246246 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:34.246607 (+ 361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47278: BlockingRecv error: recv got EOF from 127.0.0.1:47278 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 21:58:34.264919 25512 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:58:34.264227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47292 (local address 127.2.182.125:42329)
0928 21:58:34.264273 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:34.264276 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:34.264419 (+ 143us) server_negotiation.cc:407] Connection header received
0928 21:58:34.264471 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:34.264473 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:34.264485 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:34.264531 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:34.264855 (+ 324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47292: BlockingRecv error: recv got EOF from 127.0.0.1:47292 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260928 21:58:34.298669 25512 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:58:34.297994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47304 (local address 127.2.182.125:42329)
0928 21:58:34.298044 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:34.298046 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:34.298154 (+ 108us) server_negotiation.cc:407] Connection header received
0928 21:58:34.298208 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:34.298208 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:34.298221 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:34.298266 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:34.298600 (+ 334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47304: BlockingRecv error: recv got EOF from 127.0.0.1:47304 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260928 21:58:34.364696 25512 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:58:34.363888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47320 (local address 127.2.182.125:42329)
0928 21:58:34.363946 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:34.363948 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:34.364123 (+ 175us) server_negotiation.cc:407] Connection header received
0928 21:58:34.364158 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:34.364160 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:34.364172 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:34.364228 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:34.364631 (+ 403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47320: BlockingRecv error: recv got EOF from 127.0.0.1:47320 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260928 21:58:34.494720 25512 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:58:34.493890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47336 (local address 127.2.182.125:42329)
0928 21:58:34.493945 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:34.493948 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:34.494100 (+ 152us) server_negotiation.cc:407] Connection header received
0928 21:58:34.494135 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:34.494136 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:34.494149 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:34.494214 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:34.494663 (+ 449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47336: BlockingRecv error: recv got EOF from 127.0.0.1:47336 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260928 21:58:34.752775 25512 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:58:34.752176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47350 (local address 127.2.182.125:42329)
0928 21:58:34.752237 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:34.752239 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:34.752250 (+ 11us) server_negotiation.cc:407] Connection header received
0928 21:58:34.752300 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:34.752301 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:34.752311 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:34.752359 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:58:34.752713 (+ 354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47350: BlockingRecv error: recv got EOF from 127.0.0.1:47350 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 21:58:35.077374 25600 catalog_manager.cc:1259] Loaded cluster ID: 58eecb7602d646dbb84b22c1d498ae0d
I20260928 21:58:35.077458 25600 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 36a53a3cc7ca436d9469898ebb081364: loading cluster ID for follower catalog manager: success
I20260928 21:58:35.078231 25600 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 36a53a3cc7ca436d9469898ebb081364: acquiring CA information for follower catalog manager: success
I20260928 21:58:35.078480 25600 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 36a53a3cc7ca436d9469898ebb081364: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 21:58:35.139235 25644 catalog_manager.cc:1259] Loaded cluster ID: 58eecb7602d646dbb84b22c1d498ae0d
I20260928 21:58:35.139313 25644 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 27a1e245c6214921b3169f0b3b16be22: loading cluster ID for follower catalog manager: success
I20260928 21:58:35.140161 25644 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 27a1e245c6214921b3169f0b3b16be22: acquiring CA information for follower catalog manager: success
I20260928 21:58:35.140468 25644 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 27a1e245c6214921b3169f0b3b16be22: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 21:58:35.268810 25715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:35.266150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47352 (local address 127.2.182.125:42329)
0928 21:58:35.266296 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:35.266298 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:35.266308 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:58:35.266333 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:35.266334 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:35.266382 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:35.266459 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:58:35.266657 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:35.267206 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:35.268016 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:35.268236 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:35.268307 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:35.268460 (+ 153us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:58:35.268569 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:58:35.268687 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:35.268706 (+ 19us) server_negotiation.cc:299] Negotiation successful
0928 21:58:35.268725 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":68,"threads_started":1}
I20260928 21:58:35.273069 25721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:35.270600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49342 (local address 127.2.182.124:40307)
0928 21:58:35.270695 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:35.270697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:35.270704 (+ 7us) server_negotiation.cc:407] Connection header received
0928 21:58:35.270738 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:35.270739 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:35.270769 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:35.270834 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:58:35.270946 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:35.271468 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:35.272242 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:35.272438 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:35.272563 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:35.272742 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:58:35.272849 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:58:35.272909 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:35.272963 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 21:58:35.272982 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73,"thread_start_us":44,"threads_started":1}
I20260928 21:58:35.277792 25715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:35.274942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47366 (local address 127.2.182.125:42329)
0928 21:58:35.274981 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:35.274983 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:35.275077 (+ 94us) server_negotiation.cc:407] Connection header received
0928 21:58:35.275195 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:35.275196 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:35.275209 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:35.275266 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:58:35.275459 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:35.276216 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:35.277064 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:35.277340 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:35.277404 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:35.277556 (+ 152us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:58:35.277588 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:58:35.277634 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:35.277692 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 21:58:35.277713 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 21:58:35.277971 25721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:35.275150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49356 (local address 127.2.182.124:40307)
0928 21:58:35.275184 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:35.275186 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:35.275283 (+ 97us) server_negotiation.cc:407] Connection header received
0928 21:58:35.275360 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:35.275361 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:35.275371 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:35.275409 (+ 38us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:58:35.275549 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:35.276097 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:35.277161 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:35.277429 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:35.277481 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:35.277712 (+ 231us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:58:35.277745 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:58:35.277773 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:35.277871 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 21:58:35.277893 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 21:58:35.278524 25727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:58:35.274796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49524 (local address 127.2.182.126:45719)
0928 21:58:35.274910 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 21:58:35.274912 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:58:35.275036 (+ 124us) server_negotiation.cc:407] Connection header received
0928 21:58:35.275083 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:58:35.275084 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:58:35.275136 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:58:35.275214 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:58:35.275319 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:35.276184 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:35.277516 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:58:35.277794 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:58:35.277848 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:58:35.278233 (+ 385us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:58:35.278325 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:58:35.278384 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 21:58:35.278428 (+ 44us) server_negotiation.cc:299] Negotiation successful
0928 21:58:35.278450 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":56,"threads_started":1}
I20260928 21:58:35.280112 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 25365
I20260928 21:58:35.284911 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 25440
I20260928 21:58:35.289562 2777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZMcOxj/build/release/bin/kudu with pid 25515
2026-09-28T21:58:35Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2764 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5531 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (275239 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST